2024-03-24T11:01:57.420 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T11:01:57.420 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882 --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882/orig.config.yaml 2024-03-24T11:01:57.456 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:01:57.538 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617882' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGjCNhajPt4OOjUC4uIfGqvc5ymEf2cKtxBWraVxdoQw80xQOQ0+Ljj+lis9NYcREX+7b2JsfDZM89W6aIVCts= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARFtn7MZU/LuZvg4bgTANeO9ccOZvJLbhWtvvvNMk4/e0bWXTXFpmijjuFDaXC7u3kfMsUNxuOcuKZrbFKwFOs= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T11:01:57.538 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T11:01:57.539 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T11:01:57.539 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T11:01:57.541 INFO:teuthology.task.internal:Checking packages... 2024-03-24T11:01:57.571 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T11:01:57.571 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:01:57.571 INFO:teuthology.packaging:ref: None 2024-03-24T11:01:57.571 INFO:teuthology.packaging:tag: None 2024-03-24T11:01:57.572 INFO:teuthology.packaging:branch: squid 2024-03-24T11:01:57.572 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:01:57.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T11:01:57.803 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T11:01:57.805 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T11:01:57.812 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T11:01:57.812 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T11:01:57.886 INFO:teuthology.task.internal:Saving configuration 2024-03-24T11:01:57.899 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T11:01:57.905 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:01:57.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:56:55.251715', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGjCNhajPt4OOjUC4uIfGqvc5ymEf2cKtxBWraVxdoQw80xQOQ0+Ljj+lis9NYcREX+7b2JsfDZM89W6aIVCts='} 2024-03-24T11:01:57.944 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:56:55.250049', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARFtn7MZU/LuZvg4bgTANeO9ccOZvJLbhWtvvvNMk4/e0bWXTXFpmijjuFDaXC7u3kfMsUNxuOcuKZrbFKwFOs='} 2024-03-24T11:01:57.944 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T11:01:57.950 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T11:01:57.950 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-24T11:01:57.950 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T11:01:58.012 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1e09628f70>, signals=[15]) 2024-03-24T11:01:58.012 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T11:01:58.018 INFO:teuthology.task.internal:Opening connections... 2024-03-24T11:01:58.018 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-03-24T11:01:58.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:01:58.113 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-03-24T11:01:58.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:01:58.204 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T11:01:58.211 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-03-24T11:01:58.227 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-03-24T11:01:58.228 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-03-24T11:01:58.284 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-03-24T11:01:58.284 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-03-24T11:01:58.284 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-03-24T11:01:58.284 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:01:58.284 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-03-24T11:01:58.284 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:01:58.285 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:01:58.286 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-03-24T11:01:58.335 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-03-24T11:01:58.353 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-03-24T11:01:58.353 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-03-24T11:01:58.409 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-03-24T11:01:58.409 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:01:58.410 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:01:58.411 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:01:58.411 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-03-24T11:01:58.439 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T11:01:58.447 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T11:01:58.452 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T11:01:58.452 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:01:58.455 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:01:58.470 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T11:01:58.475 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T11:01:58.475 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:01:58.512 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:01:58.611 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T11:01:58.662 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T11:01:58.677 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T11:01:58.678 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T11:01:58.678 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:01:58.678 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:01:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:01:58.695 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-24T11:01:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:01:58.696 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-03-24T11:01:58.752 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:01:58.753 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.14.0-361.el9.x86_64 2024-03-24T11:01:58.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2024-03-24T11:01:58.754 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:01:58.754 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-361.el9.x86_64 2024-03-24T11:01:58.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-03-24T11:02:00.826 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T11:02:01.029 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T11:02:02.243 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-24T11:02:02.396 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-24T11:02:03.220 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-24T11:02:06.391 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-24T11:02:08.292 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T11:02:12.252 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-24T11:02:13.877 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-24T11:02:15.528 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-24T11:02:20.511 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 9.4 kB/s | 1.7 kB 00:00 2024-03-24T11:02:21.295 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:17 2024-03-24T11:02:22.610 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:02:22.610 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:02:22.706 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:02:22.712 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:02:22.713 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-03-24T11:02:22.713 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:02:22.713 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 76 M 2024-03-24T11:02:22.713 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 123 M 2024-03-24T11:02:22.713 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:02:22.871 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 386 kB/s | 44 kB 00:00 2024-03-24T11:02:25.180 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 6.1 MB/s | 15 MB 00:02 2024-03-24T11:02:27.379 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 6.1 MB/s | 27 MB 00:04 2024-03-24T11:02:28.130 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 6.2 MB/s | 33 MB 00:05 2024-03-24T11:02:28.130 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:02:28.130 INFO:teuthology.orchestra.run.smithi181.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-24T11:02:28.798 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:02:29.146 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:02:29.146 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:02:30.151 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:02:30.152 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:02:31.903 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 880 kB/s | 6.1 MB 00:07 2024-03-24T11:02:32.701 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:02:33.309 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:02:33.404 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:02:34.887 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 8.4 kB/s | 16 kB 00:01 2024-03-24T11:02:35.789 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-24T11:02:36.118 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:02:36.257 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:02:40.756 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-24T11:02:40.985 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:02:41.099 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:02:42.863 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:02:42.863 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:02:42.950 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:02:42.954 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:02:42.955 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:02:42.955 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:02:42.955 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:02:42.955 INFO:teuthology.orchestra.run.smithi179.stdout:Install 4 Packages 2024-03-24T11:02:42.955 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:02:42.955 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 76 M 2024-03-24T11:02:42.956 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 123 M 2024-03-24T11:02:42.956 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:02:43.107 INFO:teuthology.orchestra.run.smithi179.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 401 kB/s | 44 kB 00:00 2024-03-24T11:02:43.633 INFO:teuthology.orchestra.run.smithi179.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-24T11:02:44.258 INFO:teuthology.orchestra.run.smithi179.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-24T11:02:44.543 INFO:teuthology.orchestra.run.smithi179.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-24T11:02:44.543 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:02:44.543 INFO:teuthology.orchestra.run.smithi179.stdout:Total 48 MB/s | 76 MB 00:01 2024-03-24T11:02:45.208 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:02:45.581 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:02:45.581 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:02:45.844 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:02:46.588 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:02:46.588 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:02:49.063 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:02:49.697 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:02:49.778 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:02:52.501 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:02:52.662 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:02:57.338 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:02:57.488 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:02.247 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:15.176 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:15.176 INFO:teuthology.orchestra.run.smithi181.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:03:15.177 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:03:15.220 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:15.633 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:15.633 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:03:15.634 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:03:15.634 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:03:16.050 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:03:16.351 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:03:16.452 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:03:16.452 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:03:16.452 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:03:16.453 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:03:16.453 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:03:16.453 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T11:03:16.453 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:03:16.453 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:03:16.533 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:03:16.533 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:03:16.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 68.el9 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8264254 2024-03-24T11:03:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:03:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:03:16.570 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:03:16.570 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-03-24T11:03:16.570 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:03:16.570 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:03:16.571 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:03:16.571 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:03:16.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:03:17.200 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-03-24T11:03:18.610 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:03:18.646 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-03-24T11:03:18.648 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-03-24T11:03:18.664 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.HIJkBSyWvV 2024-03-24T11:03:18.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HIJkBSyWvV 2024-03-24T11:03:18.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.HIJkBSyWvV 2024-03-24T11:03:18.870 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.HIJkBSyWvV is 6KB 2024-03-24T11:03:18.920 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.HIJkBSyWvV 2024-03-24T11:03:18.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:03:19.006 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:03:19.006 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:03:19.006 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:03:19.006 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:03:19.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:03:19.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-03-24T11:03:31.213 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:31.214 INFO:teuthology.orchestra.run.smithi179.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:03:31.214 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:03:31.257 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:31.715 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:31.715 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:03:31.715 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:03:31.715 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:03:32.143 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:03:32.143 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:03:32.143 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:03:32.143 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:03:32.143 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:03:32.144 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:03:32.144 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:03:32.144 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:03:32.144 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:03:32.443 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:03:32.540 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:03:32.540 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:03:32.540 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:03:32.540 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:03:32.540 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:03:32.540 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T11:03:32.540 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:03:32.540 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:03:32.619 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:03:32.620 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:03:32.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2024-03-24T11:03:32.662 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.06 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 68.el9 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Group : Unspecified 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 8264254 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:03:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : builder@centos.org 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:03:32.664 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:03:32.666 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:03:32.666 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:03:32.666 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:03:33.323 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-03-24T11:03:34.713 INFO:teuthology.orchestra.run.smithi179.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:03:34.756 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-03-24T11:03:34.758 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-03-24T11:03:34.773 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.zyPrpAAOKW 2024-03-24T11:03:34.774 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zyPrpAAOKW 2024-03-24T11:03:34.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.zyPrpAAOKW 2024-03-24T11:03:34.985 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.zyPrpAAOKW is 6KB 2024-03-24T11:03:35.034 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.zyPrpAAOKW 2024-03-24T11:03:35.048 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:03:35.112 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:03:35.113 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:03:35.113 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:03:35.113 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:03:35.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:03:35.259 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-03-24T11:03:49.186 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T11:03:49.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-03-24T11:03:49.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:04:05.277 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T11:04:05.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-24T11:04:05.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:04:49.231 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:04:58.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-03-24T11:04:58.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:04:58.685 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-03-24T11:04:59.097 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-03-24T11:04:59.097 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:04:59.097 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-03-24T11:04:59.112 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:04:59.112 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:04:59.112 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:04:59.112 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:05:00.113 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T11:05:00.113 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:05:00.190 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-03-24T11:05:00.231 DEBUG:teuthology.parallel:result is None 2024-03-24T11:05:05.281 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:05:14.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-24T11:05:14.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:05:14.735 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-24T11:05:15.265 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-24T11:05:15.265 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:05:15.265 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-24T11:05:15.279 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:05:15.279 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:05:15.279 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:05:15.279 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:05:16.280 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T11:05:16.281 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:05:16.369 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-03-24T11:05:16.418 DEBUG:teuthology.parallel:result is None 2024-03-24T11:05:16.418 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:05:16.425 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:05:16.426 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:05:16.428 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:05:16.447 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:05:16.453 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:05:16.461 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:05:16.461 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:05:16.483 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:05:16.534 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:05:16.541 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:05:16.541 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:05:16.543 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:05:16.581 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:05:16.595 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:05:16.601 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:05:16.615 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:05:16.616 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:05:16.624 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:05:16.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:05:16.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:05:16.687 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:05:16.696 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:05:16.696 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:05:16.707 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:05:16.743 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:05:16.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:05:16.852 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:05:16.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:05:16.976 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:05:16.976 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:05:17.043 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:05:17.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:05:17.145 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:05:17.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:05:17.274 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:05:17.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:05:17.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-03-24T11:05:17.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-03-24T11:05:17.398 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:05:17.424 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:05:17.806 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:05:17.814 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:05:17.814 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:05:17.855 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:05:17.869 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T11:05:17.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-03-24T11:05:17.924 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-03-24T11:05:17.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-03-24T11:05:17.985 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-03-24T11:05:17.986 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:05:17.987 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-03-24T11:05:18.010 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-03-24T11:05:18.010 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-03-24T11:05:18.035 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-03-24T11:05:18.035 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi179.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:05:18.035 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:05:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:05:18.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:05:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:05:18.106 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:05:18.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:05:18.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:05:18.195 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:05:18.204 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:05:18.221 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:05:18.223 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi179.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:09:30.919 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi179.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-03-24T11:09:30.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-24T11:09:30.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:09:31.014 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-24T11:09:31.048 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-24T11:09:31.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-03-24T11:09:31.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:09:31.139 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-03-24T11:09:31.213 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-03-24T11:09:31.213 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:09:31.225 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:09:31.225 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:09:31.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:09:31.228 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:09:31.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:09:31.258 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:09:31.275 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:09:31.284 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:09:31.300 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:09:31.304 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-03-24T11:09:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-03-24T11:09:31.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-03-24T11:09:31.334 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:09:31.343 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-03-24T11:09:31.351 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:09:31.359 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:09:31.376 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:09:31.403 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:09:31.406 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:09:31.406 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-24T11:09:31.407 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.407 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.407 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.407 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.425 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:09:31.428 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:09:31.428 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-03-24T11:09:31.428 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.428 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.428 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.428 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:09:31.429 INFO:teuthology.run_tasks:Running task install... 2024-03-24T11:09:31.438 DEBUG:teuthology.task.install:project ceph 2024-03-24T11:09:31.438 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}} 2024-03-24T11:09:31.438 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:09:31.439 INFO:teuthology.task.install:Using flavor: default 2024-03-24T11:09:31.444 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T11:09:31.444 INFO:teuthology.task.install:extra packages: [] 2024-03-24T11:09:31.445 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:09:31.445 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:09:31.447 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:09:31.448 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:09:31.591 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T11:09:31.591 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T11:09:31.659 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T11:09:31.659 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T11:09:31.707 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:09:31.708 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:09:31.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:09:31.735 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:09:31.736 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:09:31.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:09:31.828 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:09:31.828 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:09:31.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:09:31.856 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:09:31.857 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:09:31.885 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:09:31.924 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:09:31.952 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-03-24T11:09:31.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-03-24T11:09:32.004 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:09:32.070 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-03-24T11:09:32.072 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-03-24T11:09:32.189 INFO:teuthology.orchestra.run.smithi181.stdout:74 files removed 2024-03-24T11:09:32.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-03-24T11:09:32.314 INFO:teuthology.orchestra.run.smithi179.stdout:74 files removed 2024-03-24T11:09:32.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw 2024-03-24T11:09:32.864 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 236 kB/s | 82 kB 00:00 2024-03-24T11:09:33.112 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 174 kB/s | 82 kB 00:00 2024-03-24T11:09:33.281 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-24T11:09:33.438 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-24T11:09:33.832 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-24T11:09:34.072 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-24T11:09:34.351 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T11:09:34.524 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T11:09:35.743 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-24T11:09:35.942 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-24T11:09:36.907 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-24T11:09:45.175 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 903 kB/s | 8.0 MB 00:09 2024-03-24T11:09:47.053 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T11:09:51.760 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.1 MB 00:01 2024-03-24T11:09:53.252 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-24T11:09:54.454 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-24T11:09:59.416 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 898 kB/s | 19 MB 00:21 2024-03-24T11:09:59.629 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-03-24T11:10:02.031 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:10:02.034 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:02.034 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T11:10:02.035 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T11:10:02.036 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-03-24T11:10:02.037 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout:Install 30 Packages 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 58 M 2024-03-24T11:10:02.038 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:10:03.055 INFO:teuthology.orchestra.run.smithi181.stdout:(1/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x86 6.8 MB/s | 5.3 MB 00:00 2024-03-24T11:10:03.105 INFO:teuthology.orchestra.run.smithi181.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 505 kB/s | 25 kB 00:00 2024-03-24T11:10:03.231 INFO:teuthology.orchestra.run.smithi181.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 5.8 MB/s | 742 kB 00:00 2024-03-24T11:10:03.339 INFO:teuthology.orchestra.run.smithi181.stdout:(4/32): libradosstriper1-19.0.0-1667.gdb0330b1. 4.5 MB/s | 498 kB 00:00 2024-03-24T11:10:03.540 INFO:teuthology.orchestra.run.smithi181.stdout:(5/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9. 7.7 MB/s | 9.8 MB 00:01 2024-03-24T11:10:03.623 INFO:teuthology.orchestra.run.smithi181.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 544 kB/s | 45 kB 00:00 2024-03-24T11:10:03.640 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-24T11:10:03.698 INFO:teuthology.orchestra.run.smithi181.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 1.7 MB/s | 131 kB 00:00 2024-03-24T11:10:03.774 INFO:teuthology.orchestra.run.smithi181.stdout:(8/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 2.1 MB/s | 161 kB 00:00 2024-03-24T11:10:03.849 INFO:teuthology.orchestra.run.smithi181.stdout:(9/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_6 10 MB/s | 5.1 MB 00:00 2024-03-24T11:10:03.874 INFO:teuthology.orchestra.run.smithi181.stdout:(10/32): python3-rados-19.0.0-1667.gdb0330b1.el 3.1 MB/s | 320 kB 00:00 2024-03-24T11:10:03.916 INFO:teuthology.orchestra.run.smithi181.stdout:(11/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 4.4 MB/s | 298 kB 00:00 2024-03-24T11:10:03.942 INFO:teuthology.orchestra.run.smithi181.stdout:(12/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.5 MB/s | 100 kB 00:00 2024-03-24T11:10:04.092 INFO:teuthology.orchestra.run.smithi181.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 604 kB/s | 106 kB 00:00 2024-03-24T11:10:04.109 INFO:teuthology.orchestra.run.smithi181.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 953 kB/s | 159 kB 00:00 2024-03-24T11:10:04.134 INFO:teuthology.orchestra.run.smithi181.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-24T11:10:04.201 INFO:teuthology.orchestra.run.smithi181.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-24T11:10:04.226 INFO:teuthology.orchestra.run.smithi181.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-24T11:10:04.252 INFO:teuthology.orchestra.run.smithi181.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-03-24T11:10:04.444 INFO:teuthology.orchestra.run.smithi181.stdout:(19/32): ceph-common-19.0.0-1667.gdb0330b1.el9. 9.2 MB/s | 20 MB 00:02 2024-03-24T11:10:04.519 INFO:teuthology.orchestra.run.smithi181.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 130 kB/s | 35 kB 00:00 2024-03-24T11:10:04.577 INFO:teuthology.orchestra.run.smithi181.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 334 kB/s | 44 kB 00:00 2024-03-24T11:10:04.603 INFO:teuthology.orchestra.run.smithi181.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 576 kB/s | 216 kB 00:00 2024-03-24T11:10:04.853 INFO:teuthology.orchestra.run.smithi181.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-03-24T11:10:05.012 INFO:teuthology.orchestra.run.smithi181.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 627 kB/s | 309 kB 00:00 2024-03-24T11:10:05.054 INFO:teuthology.orchestra.run.smithi181.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 242 kB/s | 49 kB 00:00 2024-03-24T11:10:05.170 INFO:teuthology.orchestra.run.smithi181.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 425 kB/s | 67 kB 00:00 2024-03-24T11:10:05.304 INFO:teuthology.orchestra.run.smithi181.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-24T11:10:05.341 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-24T11:10:05.821 INFO:teuthology.orchestra.run.smithi181.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-03-24T11:10:05.930 INFO:teuthology.orchestra.run.smithi181.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 957 kB/s | 838 kB 00:00 2024-03-24T11:10:06.172 INFO:teuthology.orchestra.run.smithi181.stdout:(30/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 9.6 MB/s | 3.4 MB 00:00 2024-03-24T11:10:06.263 INFO:teuthology.orchestra.run.smithi181.stdout:(31/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 9.4 MB/s | 3.1 MB 00:00 2024-03-24T11:10:06.872 INFO:teuthology.orchestra.run.smithi181.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 4.4 MB 00:02 2024-03-24T11:10:06.876 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:10:06.876 INFO:teuthology.orchestra.run.smithi181.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-24T11:10:06.910 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:10:06.947 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:10:06.947 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:10:07.434 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:10:07.435 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:10:09.040 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:10:09.131 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T11:10:09.160 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T11:10:09.194 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T11:10:09.217 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:10:09.235 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:10:09.432 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:10:09.490 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:10:09.828 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T11:10:09.879 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:10:09.955 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:10:10.043 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T11:10:10.072 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:10:10.135 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:10:10.164 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T11:10:10.202 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T11:10:10.277 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:10:10.300 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:10:10.346 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:10:10.373 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T11:10:10.481 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T11:10:10.508 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T11:10:10.548 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T11:10:10.581 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T11:10:10.609 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T11:10:10.642 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T11:10:10.952 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T11:10:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:10:11.079 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T11:10:11.154 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T11:10:11.608 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T11:10:11.633 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:10:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:10:11.806 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T11:10:11.836 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T11:10:13.966 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:10:14.008 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:10:14.574 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:10:14.611 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:10:14.682 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:10:14.682 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:10:14.682 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:10:14.682 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:14.721 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:10:24.415 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:10:24.415 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-03-24T11:10:24.415 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-03-24T11:10:24.415 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-03-24T11:10:24.415 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-03-24T11:10:24.416 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-03-24T11:10:24.416 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-03-24T11:10:24.416 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-03-24T11:10:24.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:25.321 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:10:25.346 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:10:25.346 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:10:25.346 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:10:25.346 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:10:25.346 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:10:25.346 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:25.347 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:10:25.433 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:10:25.433 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:10:26.540 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:10:26.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T11:10:26.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T11:10:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T11:10:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T11:10:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T11:10:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T11:10:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:10:26.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T11:10:26.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T11:10:27.755 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T11:10:27.756 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.757 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:27.758 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:10:28.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-03-24T11:10:28.645 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:29 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:10:28.792 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:10:28.793 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:28.793 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:10:28.793 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:28.793 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:28.794 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-03-24T11:10:28.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:28.795 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 48 M 2024-03-24T11:10:28.795 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 205 M 2024-03-24T11:10:28.795 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:10:29.074 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-24T11:10:29.124 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T11:10:29.150 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 572 kB/s | 163 kB 00:00 2024-03-24T11:10:29.175 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T11:10:29.367 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 864 kB/s | 187 kB 00:00 2024-03-24T11:10:29.935 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 866 kB/s | 21 MB 00:24 2024-03-24T11:10:31.151 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 21 MB/s | 47 MB 00:02 2024-03-24T11:10:31.154 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:10:31.154 INFO:teuthology.orchestra.run.smithi181.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-24T11:10:31.168 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:10:31.190 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:10:31.190 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:10:31.565 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:10:31.565 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:10:32.154 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:10:32.234 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:10:32.278 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:10:32.310 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:10:32.351 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:10:32.379 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:10:34.757 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-24T11:10:36.885 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:10:36.961 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:10:37.151 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options 2024-03-24T11:10:37.162 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T11:10:37.163 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T11:10:37.164 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout:Install 30 Packages 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 2 Packages 2024-03-24T11:10:37.165 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:10:37.166 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 58 M 2024-03-24T11:10:37.166 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:10:38.191 INFO:teuthology.orchestra.run.smithi179.stdout:(1/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x86 8.7 MB/s | 5.3 MB 00:00 2024-03-24T11:10:38.258 INFO:teuthology.orchestra.run.smithi179.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 377 kB/s | 25 kB 00:00 2024-03-24T11:10:38.350 INFO:teuthology.orchestra.run.smithi179.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-24T11:10:38.450 INFO:teuthology.orchestra.run.smithi179.stdout:(4/32): libradosstriper1-19.0.0-1667.gdb0330b1. 4.9 MB/s | 498 kB 00:00 2024-03-24T11:10:38.917 INFO:teuthology.orchestra.run.smithi179.stdout:(5/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-24T11:10:38.976 INFO:teuthology.orchestra.run.smithi179.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 777 kB/s | 45 kB 00:00 2024-03-24T11:10:39.035 INFO:teuthology.orchestra.run.smithi179.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 2.2 MB/s | 131 kB 00:00 2024-03-24T11:10:39.219 INFO:teuthology.orchestra.run.smithi179.stdout:(8/32): ceph-common-19.0.0-1667.gdb0330b1.el9.x 12 MB/s | 20 MB 00:01 2024-03-24T11:10:39.244 INFO:teuthology.orchestra.run.smithi179.stdout:(9/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 774 kB/s | 161 kB 00:00 2024-03-24T11:10:39.269 INFO:teuthology.orchestra.run.smithi179.stdout:(10/32): python3-rados-19.0.0-1667.gdb0330b1.el 6.3 MB/s | 320 kB 00:00 2024-03-24T11:10:39.294 INFO:teuthology.orchestra.run.smithi179.stdout:(11/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 5.8 MB/s | 298 kB 00:00 2024-03-24T11:10:39.320 INFO:teuthology.orchestra.run.smithi179.stdout:(12/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.9 MB/s | 100 kB 00:00 2024-03-24T11:10:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:10:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T11:10:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T11:10:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:10:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:10:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:10:39.436 INFO:teuthology.orchestra.run.smithi179.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2024-03-24T11:10:39.462 INFO:teuthology.orchestra.run.smithi179.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-24T11:10:39.487 INFO:teuthology.orchestra.run.smithi179.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-24T11:10:39.595 INFO:teuthology.orchestra.run.smithi179.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-24T11:10:39.621 INFO:teuthology.orchestra.run.smithi179.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-24T11:10:39.646 INFO:teuthology.orchestra.run.smithi179.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 822 kB/s | 41 kB 00:00 2024-03-24T11:10:39.755 INFO:teuthology.orchestra.run.smithi179.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 323 kB/s | 35 kB 00:00 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:10:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:10:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:10:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:39.799 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:10:39.839 INFO:teuthology.orchestra.run.smithi179.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 533 kB/s | 44 kB 00:00 2024-03-24T11:10:39.864 INFO:teuthology.orchestra.run.smithi179.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 894 kB/s | 216 kB 00:00 2024-03-24T11:10:40.022 INFO:teuthology.orchestra.run.smithi179.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-24T11:10:40.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-03-24T11:10:40.064 INFO:teuthology.orchestra.run.smithi179.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-03-24T11:10:40.106 INFO:teuthology.orchestra.run.smithi179.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-24T11:10:40.148 INFO:teuthology.orchestra.run.smithi179.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-24T11:10:40.315 INFO:teuthology.orchestra.run.smithi179.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-24T11:10:40.382 INFO:teuthology.orchestra.run.smithi179.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-24T11:10:40.582 INFO:teuthology.orchestra.run.smithi179.stdout:(28/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9 3.3 MB/s | 9.8 MB 00:03 2024-03-24T11:10:40.631 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:41 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:10:40.741 INFO:teuthology.orchestra.run.smithi179.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-03-24T11:10:40.790 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout:========================================================================================= 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout:========================================================================================= 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T11:10:40.793 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:10:40.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:10:40.795 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout:========================================================================================= 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout:Install 36 Packages 2024-03-24T11:10:40.796 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:40.797 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 30 M 2024-03-24T11:10:40.797 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 107 M 2024-03-24T11:10:40.797 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:10:40.866 INFO:teuthology.orchestra.run.smithi179.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-03-24T11:10:40.925 INFO:teuthology.orchestra.run.smithi179.stdout:(31/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 9.9 MB/s | 3.4 MB 00:00 2024-03-24T11:10:41.125 INFO:teuthology.orchestra.run.smithi179.stdout:(32/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 8.2 MB/s | 3.1 MB 00:00 2024-03-24T11:10:41.128 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:10:41.129 INFO:teuthology.orchestra.run.smithi179.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-24T11:10:41.161 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:10:41.199 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:10:41.199 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:10:41.678 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:10:41.678 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:10:41.686 INFO:teuthology.orchestra.run.smithi181.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-24T11:10:41.945 INFO:teuthology.orchestra.run.smithi181.stdout:(2/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 3.4 MB/s | 1.5 MB 00:00 2024-03-24T11:10:42.295 INFO:teuthology.orchestra.run.smithi181.stdout:(3/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86_ 7.7 MB/s | 4.7 MB 00:00 2024-03-24T11:10:42.337 INFO:teuthology.orchestra.run.smithi181.stdout:(4/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 2.6 MB/s | 2.1 MB 00:00 2024-03-24T11:10:42.362 INFO:teuthology.orchestra.run.smithi181.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 3.6 MB/s | 247 kB 00:00 2024-03-24T11:10:42.488 INFO:teuthology.orchestra.run.smithi181.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-24T11:10:42.513 INFO:teuthology.orchestra.run.smithi181.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-24T11:10:42.538 INFO:teuthology.orchestra.run.smithi181.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-03-24T11:10:42.563 INFO:teuthology.orchestra.run.smithi181.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-24T11:10:42.589 INFO:teuthology.orchestra.run.smithi181.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-03-24T11:10:42.614 INFO:teuthology.orchestra.run.smithi181.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T11:10:42.665 INFO:teuthology.orchestra.run.smithi181.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-24T11:10:42.690 INFO:teuthology.orchestra.run.smithi181.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-24T11:10:42.715 INFO:teuthology.orchestra.run.smithi181.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-24T11:10:42.740 INFO:teuthology.orchestra.run.smithi181.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-24T11:10:42.865 INFO:teuthology.orchestra.run.smithi181.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-24T11:10:42.907 INFO:teuthology.orchestra.run.smithi181.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 651 kB/s | 125 kB 00:00 2024-03-24T11:10:43.074 INFO:teuthology.orchestra.run.smithi181.stdout:(18/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86 15 MB/s | 17 MB 00:01 2024-03-24T11:10:43.132 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:10:43.149 INFO:teuthology.orchestra.run.smithi181.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 179 kB/s | 43 kB 00:00 2024-03-24T11:10:43.175 INFO:teuthology.orchestra.run.smithi181.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 489 kB/s | 151 kB 00:00 2024-03-24T11:10:43.213 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T11:10:43.225 INFO:teuthology.orchestra.run.smithi181.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-24T11:10:43.246 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T11:10:43.250 INFO:teuthology.orchestra.run.smithi181.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 979 kB/s | 172 kB 00:00 2024-03-24T11:10:43.267 INFO:teuthology.orchestra.run.smithi181.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-03-24T11:10:43.272 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T11:10:43.284 INFO:teuthology.orchestra.run.smithi181.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 687 kB/s | 23 kB 00:00 2024-03-24T11:10:43.303 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:10:43.309 INFO:teuthology.orchestra.run.smithi181.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 461 kB/s | 19 kB 00:00 2024-03-24T11:10:43.319 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T11:10:43.326 INFO:teuthology.orchestra.run.smithi181.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 471 kB/s | 20 kB 00:00 2024-03-24T11:10:43.360 INFO:teuthology.orchestra.run.smithi181.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-24T11:10:43.377 INFO:teuthology.orchestra.run.smithi181.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 690 kB/s | 46 kB 00:00 2024-03-24T11:10:43.402 INFO:teuthology.orchestra.run.smithi181.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-24T11:10:43.419 INFO:teuthology.orchestra.run.smithi181.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 389 kB/s | 16 kB 00:00 2024-03-24T11:10:43.453 INFO:teuthology.orchestra.run.smithi181.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-24T11:10:43.470 INFO:teuthology.orchestra.run.smithi181.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-03-24T11:10:43.474 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:10:43.495 INFO:teuthology.orchestra.run.smithi181.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-24T11:10:43.524 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T11:10:43.537 INFO:teuthology.orchestra.run.smithi181.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-24T11:10:43.563 INFO:teuthology.orchestra.run.smithi181.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-24T11:10:43.587 INFO:teuthology.orchestra.run.smithi181.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-03-24T11:10:43.591 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:10:43.591 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-24T11:10:43.616 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:10:43.658 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:10:43.658 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:10:43.855 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T11:10:43.907 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:10:43.982 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:10:44.071 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T11:10:44.076 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:10:44.076 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:10:44.108 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:10:44.154 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T11:10:44.183 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T11:10:44.227 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T11:10:44.296 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:10:44.319 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:10:44.364 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T11:10:44.392 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T11:10:44.502 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T11:10:44.543 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T11:10:44.584 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T11:10:44.616 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T11:10:44.645 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T11:10:44.678 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T11:10:44.997 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T11:10:45.033 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:10:45.106 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T11:10:45.137 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T11:10:45.179 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T11:10:45.453 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:10:45.524 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T11:10:45.556 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T11:10:45.584 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T11:10:45.638 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T11:10:45.654 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T11:10:45.669 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:10:45.700 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T11:10:45.724 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T11:10:45.732 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T11:10:45.791 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T11:10:45.821 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T11:10:45.825 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T11:10:45.902 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T11:10:45.943 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T11:10:46.015 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T11:10:46.078 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T11:10:46.116 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T11:10:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T11:10:46.229 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T11:10:46.414 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T11:10:46.462 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T11:10:46.500 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T11:10:46.535 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T11:10:46.565 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T11:10:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T11:10:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:10:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T11:10:46.749 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T11:10:46.786 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T11:10:46.819 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T11:10:46.909 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T11:10:46.949 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T11:10:47.021 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T11:10:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T11:10:47.283 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:10:47.450 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T11:10:47.479 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:10:47.505 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:10:47.505 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:10:47.505 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:10:47.505 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:10:47.506 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:10:47.506 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:48.017 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:10:48.051 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:10:48.612 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T11:10:48.646 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:10:48.722 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T11:10:48.722 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:10:48.722 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:10:48.723 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:10:48.772 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:10:48.939 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:10:48.965 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:10:48.966 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:10:48.966 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:10:48.966 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:10:48.966 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:10:48.966 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:49.406 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:10:49.429 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:10:49.430 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:10:49.430 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:10:49.430 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:10:49.430 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:10:49.430 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:49.640 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:10:49.663 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:10:49.663 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:10:49.663 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:10:49.664 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:10:49.664 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:10:49.664 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:49.728 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T11:10:51.540 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T11:10:51.541 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T11:10:51.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:10:51.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T11:10:51.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T11:10:51.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T11:10:51.542 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T11:10:51.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T11:10:51.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T11:10:51.543 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T11:10:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T11:10:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:10:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T11:10:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T11:10:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T11:10:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T11:10:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T11:10:52.534 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T11:10:52.535 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:10:52.536 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:52.537 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:10:52.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-03-24T11:10:53.447 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:10:53.545 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:10:53.597 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:10:53.598 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:10:53.598 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:10:53.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-03-24T11:10:54.223 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:55 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:10:54.370 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:10:54.371 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:54.371 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:10:54.371 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:54.371 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:10:54.371 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 748 k 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 754 k 2024-03-24T11:10:54.372 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:10:54.798 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-24T11:10:54.799 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:10:54.799 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-24T11:10:54.799 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:10:54.804 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:10:54.804 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:10:54.810 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:10:54.810 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:10:54.881 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:10:55.298 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:10:55.386 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:10:55.793 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:10:56.068 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:10:56.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:56.068 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:10:56.068 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:10:56.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:56.068 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:10:56.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:10:56.836 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:57 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:10:56.990 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:10:56.991 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:56.991 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-03-24T11:10:56.991 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:56.991 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 145 k 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 436 k 2024-03-24T11:10:56.992 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:10:57.196 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 711 kB/s | 145 kB 00:00 2024-03-24T11:10:57.197 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:10:57.197 INFO:teuthology.orchestra.run.smithi181.stdout:Total 706 kB/s | 145 kB 00:00 2024-03-24T11:10:57.197 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:10:57.205 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:10:57.205 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:10:57.256 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:10:57.256 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:10:57.394 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:10:57.447 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:10:57.471 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:10:57.471 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:10:57.471 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:10:57.471 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:58.460 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:10:58.461 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:58.461 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:10:58.461 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:10:58.461 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:10:58.461 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:10:58.464 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:10:58.464 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-03-24T11:10:58.464 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-03-24T11:10:58.464 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-03-24T11:10:58.464 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-03-24T11:10:58.465 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-03-24T11:10:58.465 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2024-03-24T11:10:58.465 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-03-24T11:10:58.465 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:10:58.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-03-24T11:10:59.299 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:00 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:10:59.350 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:10:59.374 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T11:10:59.374 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:10:59.374 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:10:59.374 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:10:59.374 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:10:59.374 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:10:59.376 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:10:59.397 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:10:59.448 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:10:59.449 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:10:59.449 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:10:59.455 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T11:10:59.455 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:10:59.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:11:00.068 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:01 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:00.217 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:00.218 INFO:teuthology.orchestra.run.smithi181.stdout:====================================================================================== 2024-03-24T11:11:00.218 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout:====================================================================================== 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T11:11:00.219 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout:====================================================================================== 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-03-24T11:11:00.220 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:00.221 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 6.9 M 2024-03-24T11:11:00.221 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-03-24T11:11:00.221 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:00.501 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1667.gdb 163 kB/s | 31 kB 00:00 2024-03-24T11:11:00.526 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 70 kB/s | 15 kB 00:00 2024-03-24T11:11:00.684 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-24T11:11:00.689 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:11:00.689 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T11:11:00.689 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T11:11:00.689 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T11:11:00.689 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T11:11:00.689 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T11:11:00.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T11:11:00.691 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T11:11:00.692 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T11:11:00.692 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T11:11:00.726 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-24T11:11:00.918 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 2.9 MB/s | 1.8 MB 00:00 2024-03-24T11:11:01.110 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 101 kB/s | 19 kB 00:00 2024-03-24T11:11:01.144 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-24T11:11:01.277 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-24T11:11:01.361 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 368 kB/s | 31 kB 00:00 2024-03-24T11:11:01.436 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-03-24T11:11:01.494 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.3 MB/s | 2.0 MB 00:00 2024-03-24T11:11:01.520 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-03-24T11:11:01.523 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:01.523 INFO:teuthology.orchestra.run.smithi181.stdout:Total 5.3 MB/s | 6.9 MB 00:01 2024-03-24T11:11:01.541 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:01.558 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:01.558 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:01.729 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:01.729 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:01.765 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T11:11:01.765 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T11:11:01.766 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:01.768 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:02.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-test 2024-03-24T11:11:02.243 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:02.321 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T11:11:02.356 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T11:11:02.402 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T11:11:02.433 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T11:11:02.526 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T11:11:02.649 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T11:11:02.678 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:28 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:02.746 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T11:11:02.795 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T11:11:02.824 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:11:02.825 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:02.826 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:02.826 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:11:02.826 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-24T11:11:02.827 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:02.828 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 48 M 2024-03-24T11:11:02.828 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 205 M 2024-03-24T11:11:02.828 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:02.856 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T11:11:03.078 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-24T11:11:03.096 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T11:11:03.123 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:11:03.128 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T11:11:03.162 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T11:11:03.279 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 428 kB/s | 163 kB 00:00 2024-03-24T11:11:03.321 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-24T11:11:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:11:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T11:11:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T11:11:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T11:11:03.847 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T11:11:04.185 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T11:11:04.185 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:04.185 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:04.185 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T11:11:04.186 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:04.186 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:04.186 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:04.186 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T11:11:04.186 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T11:11:04.187 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T11:11:04.187 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T11:11:04.187 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T11:11:04.187 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T11:11:04.187 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:11:04.187 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:11:04.188 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:04.188 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:04.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:11:05.016 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:06 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:05.168 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:05.169 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-03-24T11:11:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:11:05.169 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-03-24T11:11:05.169 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T11:11:05.169 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:11:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:05.171 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:05.171 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================ 2024-03-24T11:11:05.171 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-03-24T11:11:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:05.171 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 37 M 2024-03-24T11:11:05.171 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 209 M 2024-03-24T11:11:05.172 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:05.237 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-24T11:11:05.240 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:05.241 INFO:teuthology.orchestra.run.smithi179.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-24T11:11:05.255 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:05.276 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:05.276 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:05.418 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 235 kB/s | 32 kB 00:00 2024-03-24T11:11:05.468 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 345 kB/s | 17 kB 00:00 2024-03-24T11:11:05.552 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T11:11:05.577 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 320 kB/s | 34 kB 00:00 2024-03-24T11:11:05.660 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:05.661 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:05.744 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-24T11:11:05.811 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-24T11:11:05.836 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-24T11:11:05.945 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-16 11 MB/s | 7.4 MB 00:00 2024-03-24T11:11:06.028 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 195 kB 00:00 2024-03-24T11:11:06.070 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-03-24T11:11:06.238 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-24T11:11:06.281 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:06.362 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:11:06.411 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:11:06.446 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:11:06.496 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:11:06.524 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:11:07.046 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 242 kB/s | 245 kB 00:01 2024-03-24T11:11:07.049 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:07.050 INFO:teuthology.orchestra.run.smithi181.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-24T11:11:07.094 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:07.120 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:07.120 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:07.470 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:07.470 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:08.190 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:08.269 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:11:08.300 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:11:08.394 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:11:08.437 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:11:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:11:08.644 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:11:08.729 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:11:09.386 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:11:09.498 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:11:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:11:11.008 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T11:11:11.090 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:11:11.809 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:11:11.835 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:11:13.539 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T11:11:13.540 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T11:11:13.540 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T11:11:13.540 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:11:13.540 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:11:13.540 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:11:13.899 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:13.900 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:14.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph 2024-03-24T11:11:14.390 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:11:14.391 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:11:14.780 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:40 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:11:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:14.852 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:14.853 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:14.947 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:14.950 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-24T11:11:14.950 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:11:14.950 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-24T11:11:14.950 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:14.950 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T11:11:14.950 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:11:14.951 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:11:14.952 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-24T11:11:14.953 INFO:teuthology.orchestra.run.smithi179.stdout:Install 36 Packages 2024-03-24T11:11:14.954 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:14.954 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 30 M 2024-03-24T11:11:14.954 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 107 M 2024-03-24T11:11:14.954 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:15.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:11:15.786 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:16 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:15.858 INFO:teuthology.orchestra.run.smithi179.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 26 kB/s | 6.6 kB 00:00 2024-03-24T11:11:15.937 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:11:15.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:15.939 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-03-24T11:11:15.940 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-03-24T11:11:15.940 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:15.940 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.0 M 2024-03-24T11:11:15.940 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-03-24T11:11:15.940 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:15.982 INFO:teuthology.orchestra.run.smithi179.stdout:(2/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-03-24T11:11:16.127 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 152 kB/s | 18 kB 00:00 2024-03-24T11:11:16.152 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2024-03-24T11:11:16.203 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-24T11:11:16.208 INFO:teuthology.orchestra.run.smithi179.stdout:(3/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 2.5 MB/s | 1.5 MB 00:00 2024-03-24T11:11:16.228 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T11:11:16.253 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-24T11:11:16.279 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-24T11:11:16.316 INFO:teuthology.orchestra.run.smithi179.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 2.2 MB/s | 247 kB 00:00 2024-03-24T11:11:16.346 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-03-24T11:11:16.371 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 275 kB/s | 32 kB 00:00 2024-03-24T11:11:16.425 INFO:teuthology.orchestra.run.smithi179.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-24T11:11:16.446 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.0 MB/s | 203 kB 00:00 2024-03-24T11:11:16.467 INFO:teuthology.orchestra.run.smithi179.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T11:11:16.488 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-24T11:11:16.492 INFO:teuthology.orchestra.run.smithi179.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-24T11:11:16.514 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el 98 kB/s | 49 kB 00:00 2024-03-24T11:11:16.526 INFO:teuthology.orchestra.run.smithi179.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-24T11:11:16.538 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-24T11:11:16.559 INFO:teuthology.orchestra.run.smithi179.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-24T11:11:16.589 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-03-24T11:11:16.591 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:16.591 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.1 MB/s | 2.0 MB 00:00 2024-03-24T11:11:16.591 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:16.605 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:16.605 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:16.609 INFO:teuthology.orchestra.run.smithi179.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T11:11:16.677 INFO:teuthology.orchestra.run.smithi179.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-24T11:11:16.696 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:16.696 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:16.868 INFO:teuthology.orchestra.run.smithi179.stdout:(12/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86 19 MB/s | 17 MB 00:00 2024-03-24T11:11:16.894 INFO:teuthology.orchestra.run.smithi179.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 475 kB/s | 103 kB 00:00 2024-03-24T11:11:16.919 INFO:teuthology.orchestra.run.smithi179.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-24T11:11:16.944 INFO:teuthology.orchestra.run.smithi179.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-24T11:11:17.119 INFO:teuthology.orchestra.run.smithi179.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-24T11:11:17.214 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:17.336 INFO:teuthology.orchestra.run.smithi179.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 697 kB/s | 151 kB 00:00 2024-03-24T11:11:17.378 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:11:17.387 INFO:teuthology.orchestra.run.smithi179.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 868 kB/s | 43 kB 00:00 2024-03-24T11:11:17.440 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:11:17.462 INFO:teuthology.orchestra.run.smithi179.stdout:(19/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86 2.9 MB/s | 4.7 MB 00:01 2024-03-24T11:11:17.487 INFO:teuthology.orchestra.run.smithi179.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-24T11:11:17.495 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:11:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:11:17.571 INFO:teuthology.orchestra.run.smithi179.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 176 kB/s | 11 kB 00:00 2024-03-24T11:11:17.574 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:11:17.667 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:11:17.696 INFO:teuthology.orchestra.run.smithi179.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-24T11:11:17.721 INFO:teuthology.orchestra.run.smithi179.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 118 kB/s | 18 kB 00:00 2024-03-24T11:11:17.755 INFO:teuthology.orchestra.run.smithi179.stdout:(24/36): python3-requests-2.25.1-8.el9.noarch.r 149 kB/s | 125 kB 00:00 2024-03-24T11:11:17.776 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:11:17.780 INFO:teuthology.orchestra.run.smithi179.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 333 kB/s | 19 kB 00:00 2024-03-24T11:11:17.825 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:11:17.830 INFO:teuthology.orchestra.run.smithi179.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 172 kB/s | 23 kB 00:00 2024-03-24T11:11:17.889 INFO:teuthology.orchestra.run.smithi179.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-24T11:11:17.914 INFO:teuthology.orchestra.run.smithi179.stdout:(28/36): python3-jaraco-text-3.2.0-6.el9.noarch 123 kB/s | 20 kB 00:00 2024-03-24T11:11:17.956 INFO:teuthology.orchestra.run.smithi179.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 303 kB/s | 46 kB 00:00 2024-03-24T11:11:17.982 INFO:teuthology.orchestra.run.smithi179.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-24T11:11:18.007 INFO:teuthology.orchestra.run.smithi179.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-24T11:11:18.040 INFO:teuthology.orchestra.run.smithi179.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-03-24T11:11:18.066 INFO:teuthology.orchestra.run.smithi179.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-24T11:11:18.141 INFO:teuthology.orchestra.run.smithi179.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-24T11:11:18.166 INFO:teuthology.orchestra.run.smithi179.stdout:(35/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 429 kB/s | 90 kB 00:00 2024-03-24T11:11:18.191 INFO:teuthology.orchestra.run.smithi179.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 159 kB/s | 20 kB 00:00 2024-03-24T11:11:18.197 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:18.197 INFO:teuthology.orchestra.run.smithi179.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-03-24T11:11:18.218 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:18.256 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:18.256 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:18.547 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:11:18.615 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:11:18.648 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:11:18.664 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:18.665 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:18.696 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:11:18.718 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:11:19.462 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:11:19.463 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:11:19.463 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:11:19.463 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:11:19.463 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:11:19.817 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:11:19.817 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:19.817 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:11:19.818 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:11:19.819 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:19.819 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:19.998 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:20.061 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T11:11:20.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:11:20.093 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T11:11:20.121 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T11:11:20.192 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T11:11:20.237 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T11:11:20.270 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T11:11:20.361 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T11:11:20.448 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T11:11:20.489 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T11:11:20.561 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T11:11:20.615 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T11:11:20.642 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:21 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:20.653 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T11:11:20.711 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T11:11:20.770 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T11:11:20.793 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:20.793 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-03-24T11:11:20.793 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-03-24T11:11:20.794 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-03-24T11:11:20.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:20.795 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 6.8 M 2024-03-24T11:11:20.795 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-03-24T11:11:20.795 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:20.961 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T11:11:21.016 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T11:11:21.061 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T11:11:21.097 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T11:11:21.127 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T11:11:21.203 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T11:11:21.207 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-03-24T11:11:21.238 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:11:21.271 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T11:11:21.311 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T11:11:21.346 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T11:11:21.366 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-24T11:11:21.372 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T11:11:21.462 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T11:11:21.466 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 364 kB/s | 138 kB 00:00 2024-03-24T11:11:21.503 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T11:11:21.575 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T11:11:21.675 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 188 kB/s | 58 kB 00:00 2024-03-24T11:11:21.711 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T11:11:21.716 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-24T11:11:21.758 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2024-03-24T11:11:21.761 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:21.761 INFO:teuthology.orchestra.run.smithi181.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-03-24T11:11:21.774 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:21.784 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:21.784 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:21.830 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:11:21.873 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:21.874 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:21.995 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T11:11:22.025 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:11:22.051 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T11:11:22.051 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:11:22.051 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:11:22.051 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:11:22.051 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:11:22.051 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:22.168 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:22.340 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:11:22.393 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:11:22.919 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:11:22.996 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:11:23.066 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:11:23.089 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:11:23.493 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:11:23.519 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T11:11:23.519 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:11:23.519 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:11:23.519 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:11:23.519 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:11:23.519 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:23.850 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:11:23.850 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T11:11:23.850 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:11:23.850 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:11:23.851 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:11:23.851 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:11:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:11:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T11:11:23.992 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:11:23.992 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:11:23.992 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:11:23.992 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:11:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:24.141 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:11:24.141 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:24.141 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:24.141 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:24.141 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:11:24.141 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:11:24.141 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:11:24.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:11:24.142 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:11:24.142 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:24.142 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:24.202 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:11:24.225 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T11:11:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:11:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:11:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:11:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:11:24.226 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:24.291 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:11:24.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-03-24T11:11:24.932 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:25.081 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:25.081 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:25.081 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-03-24T11:11:25.082 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:25.083 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 964 k 2024-03-24T11:11:25.083 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.8 M 2024-03-24T11:11:25.083 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:25.210 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 621 kB/s | 79 kB 00:00 2024-03-24T11:11:25.526 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 2.0 MB/s | 886 kB 00:00 2024-03-24T11:11:25.527 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:25.528 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-24T11:11:25.528 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:25.548 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:25.548 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:25.621 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:25.622 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:25.802 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:25.935 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:11:26.001 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T11:11:26.098 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T11:11:26.099 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T11:11:26.100 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T11:11:26.100 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T11:11:26.100 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T11:11:26.100 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T11:11:26.100 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T11:11:26.101 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T11:11:26.101 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T11:11:26.101 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T11:11:26.101 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T11:11:26.101 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T11:11:26.101 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T11:11:26.102 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T11:11:26.102 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T11:11:26.102 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T11:11:26.102 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T11:11:26.710 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:11:26.710 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T11:11:27.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:11:27.022 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:27.022 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:27.022 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:11:27.022 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:27.022 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:27.136 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T11:11:27.136 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:27.136 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:27.136 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:27.136 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:11:27.137 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:11:27.138 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:27.139 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:27.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-03-24T11:11:27.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base 2024-03-24T11:11:27.871 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:28.026 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:28.027 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:28.027 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-03-24T11:11:28.027 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:28.027 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:28.027 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T11:11:28.027 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-03-24T11:11:28.027 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T11:11:28.028 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:28.028 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:28.028 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:28.028 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-03-24T11:11:28.028 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:28.028 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 332 k 2024-03-24T11:11:28.028 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.5 M 2024-03-24T11:11:28.029 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:28.035 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:28.131 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:28.150 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 568 kB/s | 69 kB 00:00 2024-03-24T11:11:28.183 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:28.184 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:11:28.184 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:28.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install cephadm 2024-03-24T11:11:28.425 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 666 kB/s | 264 kB 00:00 2024-03-24T11:11:28.426 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:28.427 INFO:teuthology.orchestra.run.smithi181.stdout:Total 835 kB/s | 332 kB 00:00 2024-03-24T11:11:28.427 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:28.435 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:28.435 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:28.478 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:28.478 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:28.626 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:28.789 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T11:11:28.806 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:28.827 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:11:28.843 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:11:28.843 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:11:28.843 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T11:11:28.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:28.955 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:28.956 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:28.956 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:11:28.956 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:28.956 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 748 k 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 754 k 2024-03-24T11:11:28.957 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:29.386 INFO:teuthology.orchestra.run.smithi179.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-24T11:11:29.386 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:29.387 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-24T11:11:29.387 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:29.391 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:29.392 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:29.397 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:29.469 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:29.582 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T11:11:29.922 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T11:11:29.923 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:29.923 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:29.923 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:29.923 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T11:11:29.923 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:29.923 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:29.926 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:11:29.999 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:11:30.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-03-24T11:11:30.401 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:11:30.672 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T11:11:30.672 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:30.673 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:30.673 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:30.673 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:30.673 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:30.813 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:31 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:30.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:11:30.963 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:30.964 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:30.964 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-03-24T11:11:30.964 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:30.964 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 127 k 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 456 k 2024-03-24T11:11:30.965 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:31.306 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 372 kB/s | 127 kB 00:00 2024-03-24T11:11:31.307 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:31.307 INFO:teuthology.orchestra.run.smithi181.stdout:Total 370 kB/s | 127 kB 00:00 2024-03-24T11:11:31.307 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:31.312 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:31.312 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:31.368 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:31.368 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:31.453 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:57 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:31.515 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:31.607 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 145 k 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 436 k 2024-03-24T11:11:31.608 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:31.613 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:31.825 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 672 kB/s | 145 kB 00:00 2024-03-24T11:11:31.826 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:31.826 INFO:teuthology.orchestra.run.smithi179.stdout:Total 667 kB/s | 145 kB 00:00 2024-03-24T11:11:31.826 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:31.833 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:31.834 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:31.885 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:31.885 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:32.025 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:32.093 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:11:32.099 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:32.117 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:11:32.117 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:11:32.117 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:11:32.118 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:32.380 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:32.380 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:32.380 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:32.380 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:32.381 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:32.381 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:32.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-03-24T11:11:33.121 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T11:11:33.122 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:33.122 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:33.122 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:33.195 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:33.297 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:33.350 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:33.351 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:11:33.351 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:33.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr 2024-03-24T11:11:33.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-03-24T11:11:33.951 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:59 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:33.976 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:34.050 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:34.101 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:34.102 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:11:34.102 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:34.127 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-03-24T11:11:34.128 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:34.129 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 31 k 2024-03-24T11:11:34.129 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 140 k 2024-03-24T11:11:34.129 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:34.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:11:34.292 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 191 kB/s | 31 kB 00:00 2024-03-24T11:11:34.293 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:34.293 INFO:teuthology.orchestra.run.smithi181.stdout:Total 189 kB/s | 31 kB 00:00 2024-03-24T11:11:34.293 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:34.296 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:34.296 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:34.321 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:34.321 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:34.444 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:34.540 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:34.727 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:00 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:34.879 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:34.880 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-24T11:11:34.880 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T11:11:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T11:11:34.882 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:11:34.882 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:11:34.882 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:34.882 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:34.882 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-24T11:11:34.882 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-24T11:11:34.882 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:34.883 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.9 M 2024-03-24T11:11:34.883 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-24T11:11:34.883 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:34.918 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:35.179 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:35.179 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:35.179 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:35.179 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:35.179 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:35.179 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:35.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-03-24T11:11:35.846 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1667.gdb 163 kB/s | 31 kB 00:00 2024-03-24T11:11:35.896 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:36 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:35.946 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 52 kB/s | 15 kB 00:00 2024-03-24T11:11:35.992 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:36.013 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-24T11:11:36.038 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-24T11:11:36.043 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:36.044 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:11:36.044 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:36.088 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 4.1 MB/s | 1.8 MB 00:00 2024-03-24T11:11:36.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-03-24T11:11:36.331 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-24T11:11:36.654 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:36.681 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 854 kB/s | 548 kB 00:00 2024-03-24T11:11:36.750 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:36.789 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-24T11:11:36.801 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:36.802 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:11:36.802 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:36.815 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-24T11:11:36.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-03-24T11:11:36.948 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-03-24T11:11:36.990 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-03-24T11:11:37.049 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): python3-repoze-lru-0.7-16.el9.noarch.r 119 kB/s | 31 kB 00:00 2024-03-24T11:11:37.052 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:37.052 INFO:teuthology.orchestra.run.smithi179.stdout:Total 3.2 MB/s | 6.9 MB 00:02 2024-03-24T11:11:37.070 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:37.087 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:37.087 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:37.260 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:37.260 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:37.411 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:37.509 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:37.561 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:37.562 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:11:37.562 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:37.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-03-24T11:11:37.776 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:37.849 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T11:11:37.884 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T11:11:37.931 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T11:11:37.970 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T11:11:38.071 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T11:11:38.170 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:38.194 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T11:11:38.267 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:38.289 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T11:11:38.319 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:38.320 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:11:38.320 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:38.333 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T11:11:38.362 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:11:38.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-03-24T11:11:38.393 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T11:11:38.637 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T11:11:38.668 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:11:38.924 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:39.021 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:39.072 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:39.073 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:11:39.073 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:39.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-03-24T11:11:39.528 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T11:11:39.528 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T11:11:39.528 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T11:11:39.528 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T11:11:39.528 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T11:11:39.528 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T11:11:39.529 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T11:11:39.529 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T11:11:39.529 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T11:11:39.529 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T11:11:39.529 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T11:11:39.529 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T11:11:39.691 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:39.788 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:11:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-03-24T11:11:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:39.876 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T11:11:39.876 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:39.876 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:39.876 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T11:11:39.876 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T11:11:39.877 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T11:11:39.878 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:11:39.878 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:11:39.878 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:39.878 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:39.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-03-24T11:11:40.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:11:40.447 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:40.596 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:40.596 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:40.596 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 85 k 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 231 k 2024-03-24T11:11:40.597 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:40.681 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:06 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:40.832 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:40.833 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-24T11:11:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:11:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:40.835 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 37 M 2024-03-24T11:11:40.836 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 209 M 2024-03-24T11:11:40.836 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:40.911 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 271 kB/s | 85 kB 00:00 2024-03-24T11:11:40.912 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:40.912 INFO:teuthology.orchestra.run.smithi181.stdout:Total 270 kB/s | 85 kB 00:00 2024-03-24T11:11:40.912 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:40.919 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:40.920 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:40.975 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:40.976 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:41.064 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 224 kB/s | 32 kB 00:00 2024-03-24T11:11:41.097 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-24T11:11:41.121 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:41.139 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 851 kB/s | 34 kB 00:00 2024-03-24T11:11:41.198 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T11:11:41.235 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:41.373 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-24T11:11:41.441 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-24T11:11:41.466 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-24T11:11:41.614 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:41.683 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-16 9.7 MB/s | 7.4 MB 00:00 2024-03-24T11:11:41.841 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-24T11:11:41.929 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:41.930 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:41.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:41.930 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:41.950 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 738 kB/s | 195 kB 00:00 2024-03-24T11:11:41.992 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-24T11:11:42.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-03-24T11:11:42.567 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 731 kB/s | 803 kB 00:01 2024-03-24T11:11:42.569 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:42.570 INFO:teuthology.orchestra.run.smithi179.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-24T11:11:42.614 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:42.639 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:42.640 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:42.740 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:43 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:42.900 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:42.901 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-03-24T11:11:42.902 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:42.902 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.1 M 2024-03-24T11:11:42.902 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 13 M 2024-03-24T11:11:42.902 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:42.994 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:42.995 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:43.582 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 4.5 MB/s | 3.1 MB 00:00 2024-03-24T11:11:43.582 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:43.583 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.5 MB/s | 3.1 MB 00:00 2024-03-24T11:11:43.583 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:43.592 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:43.592 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:43.666 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:43.666 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:43.700 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:43.781 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:11:43.821 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:11:43.915 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:11:43.950 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:11:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:44.128 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:11:44.156 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:11:44.157 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:44.182 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:44.183 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:11:44.183 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:11:44.183 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:11:44.183 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:11:44.183 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:44.236 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:11:44.884 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:11:44.993 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:11:45.302 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:45.302 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:45.302 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:45.302 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:45.302 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:45.303 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:45.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-03-24T11:11:46.131 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:47 ago on Sun 24 Mar 2024 11:09:59 AM UTC. 2024-03-24T11:11:46.229 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:11:46.280 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-03-24T11:11:46.280 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 171 k 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 494 k 2024-03-24T11:11:46.281 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-03-24T11:11:46.627 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 494 kB/s | 171 kB 00:00 2024-03-24T11:11:46.628 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:46.628 INFO:teuthology.orchestra.run.smithi181.stdout:Total 492 kB/s | 171 kB 00:00 2024-03-24T11:11:46.628 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-03-24T11:11:46.635 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-03-24T11:11:46.636 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-03-24T11:11:46.690 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-03-24T11:11:46.691 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-03-24T11:11:46.835 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-03-24T11:11:46.925 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:47.265 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:11:47.306 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:11:47.346 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:11:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:47.634 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-03-24T11:11:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:11:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:11:47.635 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-03-24T11:11:47.890 DEBUG:teuthology.parallel:result is None 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:11:49.984 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:11:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:11:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:11:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:11:49.985 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:11:50.466 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:11:50.466 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:50.466 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:50.466 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:50.466 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:11:50.467 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:11:50.467 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:11:50.469 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:11:50.469 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:11:50.469 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:11:50.470 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:11:50.470 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:11:50.470 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:11:50.470 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:11:50.470 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:11:50.471 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:50.471 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:50.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:11:51.384 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:17 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:51.536 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:11:51.537 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout:Install 13 Packages 2024-03-24T11:11:51.538 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:51.539 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 2.0 M 2024-03-24T11:11:51.539 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-24T11:11:51.539 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:51.890 INFO:teuthology.orchestra.run.smithi179.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 18 kB 00:00 2024-03-24T11:11:51.916 INFO:teuthology.orchestra.run.smithi179.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-03-24T11:11:51.966 INFO:teuthology.orchestra.run.smithi179.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-24T11:11:51.991 INFO:teuthology.orchestra.run.smithi179.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T11:11:52.016 INFO:teuthology.orchestra.run.smithi179.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-24T11:11:52.042 INFO:teuthology.orchestra.run.smithi179.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-24T11:11:52.067 INFO:teuthology.orchestra.run.smithi179.stdout:(7/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el9 165 kB/s | 49 kB 00:00 2024-03-24T11:11:52.284 INFO:teuthology.orchestra.run.smithi179.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-03-24T11:11:52.310 INFO:teuthology.orchestra.run.smithi179.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-24T11:11:52.476 INFO:teuthology.orchestra.run.smithi179.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 355 kB/s | 59 kB 00:00 2024-03-24T11:11:52.502 INFO:teuthology.orchestra.run.smithi179.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 467 kB/s | 203 kB 00:00 2024-03-24T11:11:52.643 INFO:teuthology.orchestra.run.smithi179.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-03-24T11:11:52.810 INFO:teuthology.orchestra.run.smithi179.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-24T11:11:52.812 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:52.813 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-24T11:11:52.813 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:52.828 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:52.828 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:52.926 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:52.927 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:53.449 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:53.609 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:11:53.661 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:11:53.725 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:11:53.763 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:11:53.812 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:11:53.915 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:11:54.034 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:11:54.082 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:11:54.810 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:11:54.871 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:11:54.905 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:11:54.953 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:11:54.991 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:11:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:11:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:11:56.130 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:56.131 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:11:56.371 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:11:56.936 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:22 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:11:57.088 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:11:57.089 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-24T11:11:57.089 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:11:57.089 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-24T11:11:57.089 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:11:57.089 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T11:11:57.089 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-24T11:11:57.090 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:11:57.091 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.8 M 2024-03-24T11:11:57.091 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 33 M 2024-03-24T11:11:57.091 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:11:57.364 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-24T11:11:57.447 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-24T11:11:57.556 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 361 kB/s | 138 kB 00:00 2024-03-24T11:11:57.581 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 432 kB/s | 58 kB 00:00 2024-03-24T11:11:57.615 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 871 kB/s | 51 kB 00:00 2024-03-24T11:11:57.640 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2024-03-24T11:11:57.641 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:11:57.642 INFO:teuthology.orchestra.run.smithi179.stdout:Total 12 MB/s | 6.8 MB 00:00 2024-03-24T11:11:57.654 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:11:57.665 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:11:57.665 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:11:57.754 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:11:57.754 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:11:58.049 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:11:58.205 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:11:58.258 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:11:58.775 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:11:58.853 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:11:58.914 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:11:58.937 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:11:59.766 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T11:11:59.766 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T11:11:59.766 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:11:59.766 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:11:59.767 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:11:59.767 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:12:00.061 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:12:00.061 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:00.061 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:00.061 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:12:00.062 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:12:00.062 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:12:00.062 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:12:00.062 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:12:00.062 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:12:00.062 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:00.062 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:00.316 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-fuse 2024-03-24T11:12:00.883 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:26 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:01.034 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:01.035 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:01.036 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:12:01.037 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:01.037 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-24T11:12:01.037 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:01.037 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 964 k 2024-03-24T11:12:01.037 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.8 M 2024-03-24T11:12:01.038 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:12:01.158 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 646 kB/s | 79 kB 00:00 2024-03-24T11:12:01.475 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 2.0 MB/s | 886 kB 00:00 2024-03-24T11:12:01.476 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:12:01.477 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-24T11:12:01.477 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:12:01.497 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:12:01.497 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:12:01.572 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:12:01.572 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:12:01.757 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:12:01.893 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:12:01.959 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:12:02.647 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T11:12:02.647 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T11:12:02.936 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:12:02.936 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:02.936 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:02.936 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:12:02.937 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:02.937 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:03.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-volume 2024-03-24T11:12:03.823 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:29 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:03.977 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-24T11:12:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:03.977 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-24T11:12:03.978 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 332 k 2024-03-24T11:12:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.5 M 2024-03-24T11:12:03.979 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:12:04.096 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 587 kB/s | 69 kB 00:00 2024-03-24T11:12:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 986 kB/s | 264 kB 00:00 2024-03-24T11:12:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:12:04.248 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-24T11:12:04.248 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:12:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:12:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:12:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:12:04.295 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:12:04.440 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:12:04.619 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T11:12:04.649 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:12:04.664 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T11:12:04.664 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:12:04.665 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T11:12:04.665 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:05.383 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T11:12:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T11:12:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:05.705 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T11:12:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T11:12:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:05.705 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:06.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados-devel 2024-03-24T11:12:06.607 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:06.757 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:06.757 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:06.757 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-24T11:12:06.757 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:06.758 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:12:06.758 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T11:12:06.758 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:06.758 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:12:06.758 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:06.758 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-24T11:12:06.758 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:06.759 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 127 k 2024-03-24T11:12:06.759 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 456 k 2024-03-24T11:12:06.759 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:12:06.969 INFO:teuthology.orchestra.run.smithi179.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 603 kB/s | 127 kB 00:00 2024-03-24T11:12:06.970 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:12:06.970 INFO:teuthology.orchestra.run.smithi179.stdout:Total 598 kB/s | 127 kB 00:00 2024-03-24T11:12:06.970 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:12:06.976 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:12:06.976 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:12:07.031 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:12:07.032 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:12:07.179 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:12:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:07.699 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:07.985 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:07.985 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:07.985 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:07.985 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:12:07.985 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:07.985 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:08.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs2 2024-03-24T11:12:08.813 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:08.913 INFO:teuthology.orchestra.run.smithi179.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:12:08.967 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:08.968 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:12:08.968 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:09.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs-devel 2024-03-24T11:12:09.602 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:09.753 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:09.753 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 31 k 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 140 k 2024-03-24T11:12:09.754 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:12:09.922 INFO:teuthology.orchestra.run.smithi179.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 185 kB/s | 31 kB 00:00 2024-03-24T11:12:09.923 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:12:09.923 INFO:teuthology.orchestra.run.smithi179.stdout:Total 184 kB/s | 31 kB 00:00 2024-03-24T11:12:09.923 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:12:09.926 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:12:09.927 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:12:09.954 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:12:09.954 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:12:10.082 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:12:10.179 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:10.547 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:10.818 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:10.818 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:10.818 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:10.818 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:12:10.818 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:10.819 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:10.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados2 2024-03-24T11:12:11.542 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:11.638 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:12:11.691 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:11.691 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:12:11.692 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:11.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librbd1 2024-03-24T11:12:12.298 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:12.394 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:12:12.446 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:12.447 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:12:12.447 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:12.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rados 2024-03-24T11:12:13.066 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:13.163 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:12:13.216 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:13.217 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:12:13.217 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:13.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rgw 2024-03-24T11:12:13.824 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:13.920 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:12:13.972 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:13.973 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:12:13.973 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:14.024 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-cephfs 2024-03-24T11:12:14.580 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:14.676 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:12:14.728 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:14.729 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:12:14.729 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:14.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rbd 2024-03-24T11:12:15.344 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:15.442 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T11:12:15.491 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:15.492 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-24T11:12:15.492 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:15.540 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-fuse 2024-03-24T11:12:16.098 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:42 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:16.244 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:16.245 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:16.245 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-24T11:12:16.245 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:16.245 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:12:16.245 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T11:12:16.245 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:16.246 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:12:16.246 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:16.246 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-24T11:12:16.246 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:16.246 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 85 k 2024-03-24T11:12:16.246 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 231 k 2024-03-24T11:12:16.246 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:12:16.450 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 417 kB/s | 85 kB 00:00 2024-03-24T11:12:16.451 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:12:16.451 INFO:teuthology.orchestra.run.smithi179.stdout:Total 414 kB/s | 85 kB 00:00 2024-03-24T11:12:16.451 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:12:16.458 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:12:16.458 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:12:16.513 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:12:16.513 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:12:16.659 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:12:16.757 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:17.139 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:17.434 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:12:17.434 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:17.434 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:17.666 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-mirror 2024-03-24T11:12:18.254 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:44 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:18.419 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:18.419 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 3.1 M 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 13 M 2024-03-24T11:12:18.420 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:12:18.837 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 7.4 MB/s | 3.1 MB 00:00 2024-03-24T11:12:18.838 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:12:18.838 INFO:teuthology.orchestra.run.smithi179.stdout:Total 7.4 MB/s | 3.1 MB 00:00 2024-03-24T11:12:18.839 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:12:18.848 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:12:18.848 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:12:18.923 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:12:18.923 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:12:19.350 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:12:19.422 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:19.447 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:19.447 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:12:19.447 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:12:19.447 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:12:19.447 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:12:19.447 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:20.621 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:20.622 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:20.622 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:20.622 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:12:20.622 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:20.623 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:20.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-nbd 2024-03-24T11:12:21.442 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:47 ago on Sun 24 Mar 2024 11:10:34 AM UTC. 2024-03-24T11:12:21.593 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-24T11:12:21.594 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-24T11:12:21.595 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:21.595 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 171 k 2024-03-24T11:12:21.595 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 494 k 2024-03-24T11:12:21.595 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-24T11:12:21.833 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 718 kB/s | 171 kB 00:00 2024-03-24T11:12:21.833 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:12:21.833 INFO:teuthology.orchestra.run.smithi179.stdout:Total 714 kB/s | 171 kB 00:00 2024-03-24T11:12:21.833 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-24T11:12:21.841 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-24T11:12:21.841 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-24T11:12:21.897 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-24T11:12:21.898 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-24T11:12:22.042 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-24T11:12:22.148 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:22.611 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:22.879 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T11:12:22.879 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:22.879 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-24T11:12:22.880 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T11:12:22.880 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:12:22.880 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-24T11:12:23.094 DEBUG:teuthology.parallel:result is None 2024-03-24T11:12:23.095 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:12:23.407 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:12:23.427 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T11:12:23.428 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T11:12:23.428 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T11:12:23.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:12:23.571 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:12:23.595 INFO:teuthology.orchestra.run.smithi181.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T11:12:23.596 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T11:12:23.596 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T11:12:23.598 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T11:12:23.598 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:12:23.598 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:12:23.626 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:12:23.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:12:23.667 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T11:12:23.668 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:12:23.668 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:12:23.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:12:23.762 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:12:23.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:12:23.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:12:23.861 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T11:12:23.862 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:12:23.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:12:23.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:12:23.955 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:12:23.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:12:23.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:12:24.051 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T11:12:24.051 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:12:24.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:12:24.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:12:24.145 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:12:24.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:12:24.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:12:24.240 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:12:24.344 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'cephadm_mode': 'root'} 2024-03-24T11:12:24.345 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:12:24.345 INFO:tasks.cephadm:Cluster fsid is 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:12:24.345 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:12:24.345 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.179', 'mon.c': '[v2:172.21.15.179:3301,v1:172.21.15.179:6790]', 'mon.b': '172.21.15.181'} 2024-03-24T11:12:24.345 INFO:tasks.cephadm:First mon is mon.a on smithi179 2024-03-24T11:12:24.346 INFO:tasks.cephadm:First mgr is y 2024-03-24T11:12:24.346 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:12:24.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo hostname $(hostname -s) 2024-03-24T11:12:24.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-03-24T11:12:24.400 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T11:12:24.401 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:12:24.542 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77704/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1667-gdb0330b1', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1667.gdb0330b1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-16 11:55:36.797747', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-24T11:12:24.789 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 db0330b1e4e2470d52b750e251e55a522b4f7d69 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:12:24.791 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm 2024-03-24T11:12:24.791 DEBUG:teuthology.orchestra.run.smithi179:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:12:25.232 INFO:teuthology.orchestra.run.smithi179.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:12 /home/ubuntu/cephtest/cephadm 2024-03-24T11:12:25.232 DEBUG:teuthology.orchestra.run.smithi181:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:12:25.679 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:12 /home/ubuntu/cephtest/cephadm 2024-03-24T11:12:25.679 DEBUG:teuthology.orchestra.run.smithi179:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:12:25.696 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:12:25.726 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on all hosts... 2024-03-24T11:12:25.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:12:25.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:12:26.583 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:12:26.626 INFO:teuthology.orchestra.run.smithi179.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:13:08.909 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:13:08.910 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:13:08.910 INFO:teuthology.orchestra.run.smithi179.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:13:08.910 INFO:teuthology.orchestra.run.smithi179.stdout: "repo_digests": [ 2024-03-24T11:13:08.910 INFO:teuthology.orchestra.run.smithi179.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:13:08.910 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:13:08.910 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:13:09.662 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:13:09.662 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:13:09.662 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:13:09.662 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-03-24T11:13:09.662 INFO:teuthology.orchestra.run.smithi181.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:13:09.662 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:13:09.663 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:13:09.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph 2024-03-24T11:13:09.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-03-24T11:13:09.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /etc/ceph 2024-03-24T11:13:09.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-03-24T11:13:09.821 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:13:09.822 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-24T11:13:09.823 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-24T11:13:09.824 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-24T11:13:09.824 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T11:13:09.824 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-24T11:13:09.824 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:13:09.824 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:13:09.851 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6430538c-e9cf-11ee-95cd-87774f69a715 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-24T11:13:09.851 DEBUG:teuthology.orchestra.run.smithi179:mon.a> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T11:13:09.894 DEBUG:teuthology.orchestra.run.smithi179:mgr.y> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T11:13:09.937 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:13:09.937 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 -v bootstrap --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.179 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:13:10.130 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:13:10.130 INFO:teuthology.orchestra.run.smithi179.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69', '-v', 'bootstrap', '--fsid', '6430538c-e9cf-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.179', '--skip-admin-label'] 2024-03-24T11:13:10.154 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:13:10.155 INFO:teuthology.orchestra.run.smithi179.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T11:13:10.155 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying podman|docker is present... 2024-03-24T11:13:10.179 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:13:10.179 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying lvm2 is present... 2024-03-24T11:13:10.179 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying time synchronization is in place... 2024-03-24T11:13:10.186 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:13:10.186 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:13:10.192 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:13:10.192 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout inactive 2024-03-24T11:13:10.198 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout enabled 2024-03-24T11:13:10.204 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout active 2024-03-24T11:13:10.204 INFO:teuthology.orchestra.run.smithi179.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:13:10.205 INFO:teuthology.orchestra.run.smithi179.stdout:Repeating the final host check... 2024-03-24T11:13:10.227 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:13:10.227 INFO:teuthology.orchestra.run.smithi179.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:13:10.227 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl is present 2024-03-24T11:13:10.227 INFO:teuthology.orchestra.run.smithi179.stdout:lvcreate is present 2024-03-24T11:13:10.233 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:13:10.234 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:13:10.239 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:13:10.239 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout inactive 2024-03-24T11:13:10.245 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout enabled 2024-03-24T11:13:10.251 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout active 2024-03-24T11:13:10.251 INFO:teuthology.orchestra.run.smithi179.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:13:10.252 INFO:teuthology.orchestra.run.smithi179.stdout:Host looks OK 2024-03-24T11:13:10.252 INFO:teuthology.orchestra.run.smithi179.stdout:Cluster fsid: 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:10.252 INFO:teuthology.orchestra.run.smithi179.stdout:Acquiring lock 139652837222000 on /run/cephadm/6430538c-e9cf-11ee-95cd-87774f69a715.lock 2024-03-24T11:13:10.252 INFO:teuthology.orchestra.run.smithi179.stdout:Lock 139652837222000 acquired on /run/cephadm/6430538c-e9cf-11ee-95cd-87774f69a715.lock 2024-03-24T11:13:10.252 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying IP 172.21.15.179 port 3300 ... 2024-03-24T11:13:10.252 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying IP 172.21.15.179 port 6789 ... 2024-03-24T11:13:10.253 INFO:teuthology.orchestra.run.smithi179.stdout:Base mon IP(s) is [172.21.15.179:3300, 172.21.15.179:6789], mon addrv is [v2:172.21.15.179:3300,v1:172.21.15.179:6789] 2024-03-24T11:13:10.255 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.179 metric 100 2024-03-24T11:13:10.256 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.179 metric 100 2024-03-24T11:13:10.258 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:13:10.258 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T11:13:10.258 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-03-24T11:13:10.260 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:13:10.260 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T11:13:10.261 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:13:10.261 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T11:13:10.261 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:725b/64 scope link 2024-03-24T11:13:10.261 INFO:teuthology.orchestra.run.smithi179.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:13:10.262 INFO:teuthology.orchestra.run.smithi179.stdout:Mon IP `172.21.15.179` is in CIDR network `172.21.0.0/20` 2024-03-24T11:13:10.262 INFO:teuthology.orchestra.run.smithi179.stdout:Mon IP `172.21.15.179` is in CIDR network `172.21.0.0/20` 2024-03-24T11:13:10.262 INFO:teuthology.orchestra.run.smithi179.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T11:13:10.262 INFO:teuthology.orchestra.run.smithi179.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:13:10.262 INFO:teuthology.orchestra.run.smithi179.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:13:10.718 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stdout 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:13:10.718 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:13:10.718 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T11:13:10.718 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stderr Copying blob sha256:989de1c398e87dbcb4906791a352373dc0e0415d619bc5b36f1f1823df929cb1 2024-03-24T11:13:10.719 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-24T11:13:10.719 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stderr Copying config sha256:4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:13:10.719 INFO:teuthology.orchestra.run.smithi179.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T11:13:11.681 INFO:teuthology.orchestra.run.smithi179.stdout:ceph: stdout ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:13:11.681 INFO:teuthology.orchestra.run.smithi179.stdout:Ceph version: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:13:11.682 INFO:teuthology.orchestra.run.smithi179.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T11:13:12.546 INFO:teuthology.orchestra.run.smithi179.stdout:stat: stdout 167 167 2024-03-24T11:13:12.546 INFO:teuthology.orchestra.run.smithi179.stdout:Creating initial keys... 2024-03-24T11:13:13.526 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph-authtool: stdout AQDICgBmKm7kMhAA5wbkdhjdpwX4z596gzkArw== 2024-03-24T11:13:14.407 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph-authtool: stdout AQDJCgBmmVRCMxAAVJ+cBHiQlLyecQRYYsgK0g== 2024-03-24T11:13:15.421 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph-authtool: stdout AQDKCgBmS70lMRAADHcx5OhvvR0ftSclzdNlNg== 2024-03-24T11:13:15.422 INFO:teuthology.orchestra.run.smithi179.stdout:Creating initial monmap... 2024-03-24T11:13:16.303 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:13:16.303 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T11:13:16.303 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:16.303 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:13:16.303 INFO:teuthology.orchestra.run.smithi179.stdout:monmaptool for a [v2:172.21.15.179:3300,v1:172.21.15.179:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:13:16.303 INFO:teuthology.orchestra.run.smithi179.stdout:setting min_mon_release = quincy 2024-03-24T11:13:16.304 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/monmaptool: set fsid to 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:16.304 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:13:16.304 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:13:16.304 INFO:teuthology.orchestra.run.smithi179.stdout:Creating mon... 2024-03-24T11:13:17.152 INFO:teuthology.orchestra.run.smithi179.stdout:create mon.a on 2024-03-24T11:13:17.351 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T11:13:17.503 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:13:17.698 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6430538c-e9cf-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-6430538c-e9cf-11ee-95cd-87774f69a715.target. 2024-03-24T11:13:17.699 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6430538c-e9cf-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-6430538c-e9cf-11ee-95cd-87774f69a715.target. 2024-03-24T11:13:17.947 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:13:17.947 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T11:13:18.113 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6430538c-e9cf-11ee-95cd-87774f69a715.target.wants/ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-6430538c-e9cf-11ee-95cd-87774f69a715@.service. 2024-03-24T11:13:18.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:18 smithi179 podman[29223]: 2024-03-24 11:13:18.263144179 +0000 UTC m=+0.019190689 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:13:18.667 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:13:18.667 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout disabled 2024-03-24T11:13:18.674 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:13:18.674 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout inactive 2024-03-24T11:13:18.674 INFO:teuthology.orchestra.run.smithi179.stdout:firewalld.service is not enabled 2024-03-24T11:13:18.674 INFO:teuthology.orchestra.run.smithi179.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:13:18.675 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for mon to start... 2024-03-24T11:13:18.675 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for mon... 2024-03-24T11:13:18.749 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:18 smithi179 podman[29223]: 2024-03-24 11:13:18.569462584 +0000 UTC m=+0.325509079 container init 6658d88bce2b43dbf62c3b347f01f605c938003a6c2245ed09c1cb3a938520ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:13:18.749 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:18 smithi179 podman[29223]: 2024-03-24 11:13:18.57230629 +0000 UTC m=+0.328352787 container start 6658d88bce2b43dbf62c3b347f01f605c938003a6c2245ed09c1cb3a938520ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:13:18.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:18 smithi179 ceph-mon[29242]: mkfs 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:18.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:18 smithi179 ceph-mon[29242]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:13:18.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:18 smithi179 bash[29223]: 6658d88bce2b43dbf62c3b347f01f605c938003a6c2245ed09c1cb3a938520ef 2024-03-24T11:13:18.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:18 smithi179 systemd[1]: Started Ceph mon.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:13:19.759 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:19 smithi179 ceph-mon[29242]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:13:19.760 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:19 smithi179 ceph-mon[29242]: monmap e1: 1 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:13:19.760 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:19 smithi179 ceph-mon[29242]: fsmap 2024-03-24T11:13:19.760 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:19 smithi179 ceph-mon[29242]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:13:19.760 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:19 smithi179 ceph-mon[29242]: mgrmap e1: no daemons active 2024-03-24T11:13:19.760 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:19 smithi179 ceph-mon[29242]: from='client.? 172.21.15.179:0/1064029245' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T11:13:19.764 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T11:13:19.764 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout id: 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:19.764 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T11:13:19.764 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:19.764 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout services: 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.566537s) 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout data: 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:mon is available 2024-03-24T11:13:19.765 INFO:teuthology.orchestra.run.smithi179.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T11:13:20.792 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:20.794 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:13:20.794 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout fsid = 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:20.794 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.179:3300,v1:172.21.15.179:6789] 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:13:20.795 INFO:teuthology.orchestra.run.smithi179.stdout:Generating new minimal ceph.conf... 2024-03-24T11:13:20.899 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:20 smithi179 ceph-mon[29242]: from='client.? 172.21.15.179:0/2292129802' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:13:20.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:20 smithi179 ceph-mon[29242]: from='client.? 172.21.15.179:0/2292129802' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:13:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:21 smithi179 ceph-mon[29242]: from='client.? 172.21.15.179:0/3640325344' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:22.091 INFO:teuthology.orchestra.run.smithi179.stdout:Restarting the monitor... 2024-03-24T11:13:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:22 smithi179 systemd[1]: Stopping Ceph mon.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:13:23.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:23 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29238]: 2024-03-24T11:13:23.202+0000 7f0adf03f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:13:23.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:23 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29238]: 2024-03-24T11:13:23.202+0000 7f0adf03f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:13:23.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:23 smithi179 podman[29485]: 2024-03-24 11:13:23.416703673 +0000 UTC m=+1.160763318 container died 6658d88bce2b43dbf62c3b347f01f605c938003a6c2245ed09c1cb3a938520ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b) 2024-03-24T11:13:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:27 smithi179 podman[29485]: 2024-03-24 11:13:27.385241013 +0000 UTC m=+5.129300658 container cleanup 6658d88bce2b43dbf62c3b347f01f605c938003a6c2245ed09c1cb3a938520ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b) 2024-03-24T11:13:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:27 smithi179 bash[29485]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:13:28.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:27 smithi179 podman[29499]: 2024-03-24 11:13:27.988172843 +0000 UTC m=+4.567516091 container remove 6658d88bce2b43dbf62c3b347f01f605c938003a6c2245ed09c1cb3a938520ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T11:13:28.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T11:13:28.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 systemd[1]: Stopped Ceph mon.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:13:28.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 systemd[1]: Starting Ceph mon.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 podman[29589]: 2024-03-24 11:13:28.380484736 +0000 UTC m=+0.098860580 container create d0b2dc392a46005f899fe6011f9db7030a28614a39f1d7007feecffbe68852f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 podman[29589]: 2024-03-24 11:13:28.299242601 +0000 UTC m=+0.017618450 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 podman[29589]: 2024-03-24 11:13:28.582841022 +0000 UTC m=+0.301216865 container init d0b2dc392a46005f899fe6011f9db7030a28614a39f1d7007feecffbe68852f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 podman[29589]: 2024-03-24 11:13:28.585790292 +0000 UTC m=+0.304166139 container start d0b2dc392a46005f899fe6011f9db7030a28614a39f1d7007feecffbe68852f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: pidfile_write: ignore empty --pid-file 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: load: jerasure load: lrc 2024-03-24T11:13:28.658 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Git sha 0 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: DB SUMMARY 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: DB Session ID: B48RVGGNB1MUVGE4DBM9 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 90847 ; 2024-03-24T11:13:28.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.env: 0x55be752f3ae0 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.info_log: 0x55be7772f8c0 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:13:28.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.statistics: (nil) 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.use_fsync: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.db_log_dir: 2024-03-24T11:13:28.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.wal_dir: 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.write_buffer_manager: 0x55be77703400 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:13:28.662 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.unordered_write: 0 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.row_cache: None 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.wal_filter: None 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:13:28.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.wal_compression: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:13:28.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:13:28.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_open_files: -1 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Compression algorithms supported: 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kZSTD supported: 0 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:13:28.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.merge_operator: 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_filter: None 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55be7772f9c0) 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:13:28.667 INFO:journalctl@ceph.mon.a.smithi179.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: index_type: 0 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: data_block_index_type: 0 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: index_shortening: 1 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: checksum: 4 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: no_block_cache: 0 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_cache: 0x55be77733090 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:13:28.668 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_cache_options: 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: capacity : 536870912 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: num_shard_bits : 4 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: strict_capacity_limit : 0 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_cache_compressed: (nil) 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: persistent_cache: (nil) 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_size: 4096 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_size_deviation: 10 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_restart_interval: 16 2024-03-24T11:13:28.669 INFO:journalctl@ceph.mon.a.smithi179.stdout: index_block_restart_interval: 1 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: metadata_block_size: 4096 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: partition_filters: 0 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: use_delta_encoding: 1 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: filter_policy: bloomfilter 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: whole_key_filtering: 1 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: verify_compression: 0 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: format_version: 5 2024-03-24T11:13:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout: enable_index_compression: 1 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout: block_align: 0 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout: max_auto_readahead_size: 262144 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout: prepopulate_block_cache: 0 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression: NoCompression 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.num_levels: 7 2024-03-24T11:13:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:13:28.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:13:28.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:13:28.674 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:13:28.675 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:13:28.676 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:13:28.676 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:13:28.676 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:13:28.676 INFO:teuthology.orchestra.run.smithi179.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T11:13:28.676 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.ttl: 2592000 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:13:28.677 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 19099efd-1bbb-47be-8f1b-3950bc0a740e 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278808615913, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278808617223, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86842, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 84924, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11580, "raw_average_key_size": 48, "raw_value_size": 78545, "raw_average_value_size": 331, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711278808, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "19099efd-1bbb-47be-8f1b-3950bc0a740e", "db_session_id": "B48RVGGNB1MUVGE4DBM9", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278808617330, "job": 1, "event": "recovery_finished"} 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55be77824000 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: DB pointer 0x55be7780e000 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout: ** DB Stats ** 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.a.smithi179.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: ** Compaction Stats [default] ** 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: L0 2/0 86.64 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: Sum 2/0 86.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: ** Compaction Stats [default] ** 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:13:28.680 INFO:journalctl@ceph.mon.a.smithi179.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 104.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: Cumulative compaction: 0.00 GB write, 19.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: Interval compaction: 0.00 GB write, 19.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:13:28.681 INFO:journalctl@ceph.mon.a.smithi179.stdout: Block cache BinnedLRUCache@0x55be77733090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: starting mon.a rank 0 at public addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] at bind addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???) e1 preinit fsid 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???).mds e1 new map 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???).mds e1 print_map 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: e1 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: legacy client fscid: -1 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout: No filesystems configured 2024-03-24T11:13:28.682 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:13:28.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:13:28.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:13:28.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:13:28.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:13:28.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 bash[29589]: d0b2dc392a46005f899fe6011f9db7030a28614a39f1d7007feecffbe68852f7 2024-03-24T11:13:28.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 systemd[1]: Started Ceph mon.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:13:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:13:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: monmap e1: 1 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:13:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: fsmap 2024-03-24T11:13:28.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:13:28.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:28 smithi179 ceph-mon[29608]: mgrmap e1: no daemons active 2024-03-24T11:13:29.708 INFO:teuthology.orchestra.run.smithi179.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:13:29.730 INFO:teuthology.orchestra.run.smithi179.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:13:29.730 INFO:teuthology.orchestra.run.smithi179.stdout:Creating mgr... 2024-03-24T11:13:29.731 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T11:13:29.731 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T11:13:29.946 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:13:29.947 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T11:13:30.095 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6430538c-e9cf-11ee-95cd-87774f69a715.target.wants/ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-6430538c-e9cf-11ee-95cd-87774f69a715@.service. 2024-03-24T11:13:30.108 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:30 smithi179 systemd[1]: Starting Ceph mgr.y for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:13:30.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:30 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3212298300' entity='client.admin' 2024-03-24T11:13:30.405 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:30 smithi179 podman[29829]: 2024-03-24 11:13:30.316264064 +0000 UTC m=+0.105739970 container create e99c22ba74426d013c47c1e790fab6ebfdbbf3b82ed1abdf44bb693e3f4888ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T11:13:30.405 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:30 smithi179 podman[29829]: 2024-03-24 11:13:30.228236413 +0000 UTC m=+0.017712291 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:13:30.608 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:13:30.608 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout disabled 2024-03-24T11:13:30.615 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:13:30.616 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout inactive 2024-03-24T11:13:30.616 INFO:teuthology.orchestra.run.smithi179.stdout:firewalld.service is not enabled 2024-03-24T11:13:30.616 INFO:teuthology.orchestra.run.smithi179.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:13:30.624 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:13:30.624 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout disabled 2024-03-24T11:13:30.633 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:13:30.633 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout inactive 2024-03-24T11:13:30.633 INFO:teuthology.orchestra.run.smithi179.stdout:firewalld.service is not enabled 2024-03-24T11:13:30.633 INFO:teuthology.orchestra.run.smithi179.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T11:13:30.633 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for mgr to start... 2024-03-24T11:13:30.633 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for mgr... 2024-03-24T11:13:30.757 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:30 smithi179 podman[29829]: 2024-03-24 11:13:30.49370411 +0000 UTC m=+0.283179964 container init e99c22ba74426d013c47c1e790fab6ebfdbbf3b82ed1abdf44bb693e3f4888ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T11:13:30.757 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:30 smithi179 podman[29829]: 2024-03-24 11:13:30.496091339 +0000 UTC m=+0.285567194 container start e99c22ba74426d013c47c1e790fab6ebfdbbf3b82ed1abdf44bb693e3f4888ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T11:13:30.757 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:30 smithi179 bash[29829]: e99c22ba74426d013c47c1e790fab6ebfdbbf3b82ed1abdf44bb693e3f4888ec 2024-03-24T11:13:30.757 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:30 smithi179 systemd[1]: Started Ceph mgr.y for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:13:31.441 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:31 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3375401778' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:13:31.441 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:31.055+0000 7efe07732200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:13:31.441 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:31.154+0000 7efe07732200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:13:31.752 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:31.755 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "fsid": "6430538c-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:31.756 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:13:31.757 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:13:31.760 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:13:31.761 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:13:31.762 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:31.763 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:13:18.640435+0000", 2024-03-24T11:13:31.763 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:13:31.763 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:31.763 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:13:31.763 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:13:31.763 INFO:teuthology.orchestra.run.smithi179.stdout:mgr not available, waiting (1/15)... 2024-03-24T11:13:32.041 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:31.810+0000 7efe07732200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:13:32.041 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:31.901+0000 7efe07732200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:13:32.041 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.040+0000 7efe07732200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:13:32.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.134+0000 7efe07732200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:13:32.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.216+0000 7efe07732200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:13:32.760 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.475+0000 7efe07732200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:13:32.760 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.581+0000 7efe07732200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:13:32.760 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.666+0000 7efe07732200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:13:32.760 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.759+0000 7efe07732200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:13:33.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:32.934+0000 7efe07732200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:13:34.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:33 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:33.909+0000 7efe07732200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:13:34.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:34 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:34.039+0000 7efe07732200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:13:34.653 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:34 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:34.370+0000 7efe07732200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:13:34.653 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:34 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:34.459+0000 7efe07732200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:13:34.653 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:34 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:34.652+0000 7efe07732200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:13:34.959 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:34 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:34.837+0000 7efe07732200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:13:35.380 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:35.117+0000 7efe07732200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:13:35.381 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:35.380+0000 7efe07732200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:13:35.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:35.613+0000 7efe07732200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:13:36.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:36.412+0000 7efe07732200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:13:37.349 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:37 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3735424936' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:13:37.349 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:36.985+0000 7efe07732200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:13:37.349 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:37.088+0000 7efe07732200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:13:37.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:37.311+0000 7efe07732200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:13:38.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: Activating manager daemon y 2024-03-24T11:13:38.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: mgrmap e2: y(active, starting, since 0.00311967s) 2024-03-24T11:13:38.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:13:38.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:13:38.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:13:38.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:13:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:13:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: Manager daemon y is now available 2024-03-24T11:13:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:13:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:13:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' 2024-03-24T11:13:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' 2024-03-24T11:13:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:38 smithi179 ceph-mon[29608]: from='mgr.14100 172.21.15.179:0/4139143871' entity='mgr.y' 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "fsid": "6430538c-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:13:39.214 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:13:39.215 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:13:39.216 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:13:39.217 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:13:18.640435+0000", 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:13:39.218 INFO:teuthology.orchestra.run.smithi179.stdout:mgr not available, waiting (2/15)... 2024-03-24T11:13:39.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:39 smithi179 ceph-mon[29608]: mgrmap e3: y(active, since 1.00712s) 2024-03-24T11:13:41.625 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:41 smithi179 ceph-mon[29608]: mgrmap e4: y(active, since 3s) 2024-03-24T11:13:42.553 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:42 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4021088727' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:13:42.556 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:42.559 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:13:42.559 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "fsid": "6430538c-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:13:42.559 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:13:42.559 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:13:42.559 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:13:42.559 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:13:42.560 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:13:42.561 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:13:42.564 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:42.564 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:13:42.565 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:13:42.566 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:13:18.640435+0000", 2024-03-24T11:13:42.567 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:13:42.567 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:13:42.567 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:13:42.567 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:13:42.567 INFO:teuthology.orchestra.run.smithi179.stdout:mgr is available 2024-03-24T11:13:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:43 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4139496559' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:13:43.814 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:43.814 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:13:43.814 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout fsid = 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:43.814 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:13:43.814 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:13:43.814 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:13:43.814 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:13:43.815 INFO:teuthology.orchestra.run.smithi179.stdout:Enabling cephadm module... 2024-03-24T11:13:44.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2787153145' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:13:44.859 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:44 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ignoring --setuser ceph since I am not root 2024-03-24T11:13:44.859 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:44 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ignoring --setgroup ceph since I am not root 2024-03-24T11:13:45.110 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:45.106+0000 7f2341835200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:13:45.366 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:45.207+0000 7f2341835200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:13:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:45 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2787153145' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:13:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:45 smithi179 ceph-mon[29608]: mgrmap e5: y(active, since 7s) 2024-03-24T11:13:46.094 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:45.856+0000 7f2341835200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:13:46.094 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:45.952+0000 7f2341835200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:13:46.094 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:46.093+0000 7f2341835200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:13:46.429 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:46.189+0000 7f2341835200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:13:46.429 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:46.271+0000 7f2341835200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:13:46.433 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:13:46.433 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T11:13:46.433 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:13:46.433 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:13:46.434 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:13:46.434 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:13:46.434 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for the mgr to restart... 2024-03-24T11:13:46.434 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for mgr epoch 5... 2024-03-24T11:13:46.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/930042822' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:13:46.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:46.538+0000 7f2341835200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:13:46.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:46.651+0000 7f2341835200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:13:47.015 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:46.739+0000 7f2341835200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:13:47.015 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:46 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:46.837+0000 7f2341835200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:13:47.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:47.014+0000 7f2341835200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:13:48.425 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:47.964+0000 7f2341835200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:13:48.425 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:48 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:48.094+0000 7f2341835200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:13:48.425 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:48 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:48.424+0000 7f2341835200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:13:48.708 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:48 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:48.514+0000 7f2341835200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:13:48.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:48 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:48.708+0000 7f2341835200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:13:48.959 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:48 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:48.890+0000 7f2341835200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:13:49.432 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:49 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:49.172+0000 7f2341835200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:13:49.432 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:49 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:49.432+0000 7f2341835200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:13:49.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:49 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:49.667+0000 7f2341835200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:13:50.720 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:50 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:50.468+0000 7f2341835200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:13:51.360 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:51.045+0000 7f2341835200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:13:51.361 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:51.143+0000 7f2341835200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:13:51.361 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:13:51.360+0000 7f2341835200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: Active manager daemon y restarted 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: Activating manager daemon y 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: mgrmap e6: y(active, starting, since 0.00334008s) 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:13:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:13:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:13:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: Manager daemon y is now available 2024-03-24T11:13:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:13:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:13:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:51 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: Found migration_current of "None". Setting to last migration. 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: mgrmap e7: y(active, since 1.00536s) 2024-03-24T11:13:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:52 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:53.008 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:13:53.008 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T11:13:53.008 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:13:53.008 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:13:53.008 INFO:teuthology.orchestra.run.smithi179.stdout:mgr epoch 5 is available 2024-03-24T11:13:53.009 INFO:teuthology.orchestra.run.smithi179.stdout:Setting orchestrator backend to cephadm... 2024-03-24T11:13:53.627 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:53 smithi179 ceph-mon[29608]: [24/Mar/2024:11:13:52] ENGINE Bus STARTING 2024-03-24T11:13:53.627 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:53 smithi179 ceph-mon[29608]: [24/Mar/2024:11:13:52] ENGINE Serving on https://172.21.15.179:7150 2024-03-24T11:13:53.627 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:53 smithi179 ceph-mon[29608]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:13:53.627 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:53 smithi179 ceph-mon[29608]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:13:53.628 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:53 smithi179 ceph-mon[29608]: [24/Mar/2024:11:13:52] ENGINE Serving on http://172.21.15.179:8765 2024-03-24T11:13:53.628 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:53 smithi179 ceph-mon[29608]: [24/Mar/2024:11:13:52] ENGINE Bus STARTED 2024-03-24T11:13:54.599 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:54 smithi179 ceph-mon[29608]: mgrmap e8: y(active, since 2s) 2024-03-24T11:13:54.599 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:54 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:13:54.599 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:54 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:55.538 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T11:13:55.538 INFO:teuthology.orchestra.run.smithi179.stdout:Generating ssh key... 2024-03-24T11:13:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:55 smithi179 ceph-mon[29608]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-mon[29608]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:56.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: Generating public/private rsa key pair. 2024-03-24T11:13:56.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: Your identification has been saved in /tmp/tmpzci4s0s1/key. 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: Your public key has been saved in /tmp/tmpzci4s0s1/key.pub. 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: The key fingerprint is: 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: SHA256:rZ3wPzoO3Vetnt5GTAFUgKlRnoXsICg82ayAKXPamiI ceph-6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: The key's randomart image is: 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: +---[RSA 3072]----+ 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: |... + . o.==+. | 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: |=..= + . o.=o . | 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: |.=. + . =o .| 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: |. .. o . o| 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: | o S . oo| 2024-03-24T11:13:56.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: |E * o oo| 2024-03-24T11:13:56.711 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: |o o = . o. | 2024-03-24T11:13:56.711 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: | ....o o.| 2024-03-24T11:13:56.711 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: | .oo.o+..| 2024-03-24T11:13:56.711 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:13:56 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: +----[SHA256]-----+ 2024-03-24T11:13:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:57 smithi179 ceph-mon[29608]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:57 smithi179 ceph-mon[29608]: Generating ssh key... 2024-03-24T11:13:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:57 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:13:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:57 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:13:58.192 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/f2JVCdcpDrSHkG9kt69CP7jDodmLHL21YJI8dX30lRbwWeDHW+nViQbkF2Ivc8fc5yUe8XfGSkUuZKHJiBVmsIwUHwsN+jPXFpoQNldpmIsnf7YO/+iaFNubA593TwuOQfxjoZfMxOLYP3brrdLu2VoDeg9zfaMl/sUll+9p1TggB2rgqB/ULea+UwhQEfm+1mP2PHAU4hFb+hzk2u6GwZL/1PpIW/bstzPKd49rNQAANNBZ8FkoAqC6hjJX/BqBMgW3Olepk6YWKSsUN7EDkPrG3YioBev2Cr1BV1Dn5TUiWmIB6jccNPmAqLr/tB2vxTx/COoO4porxP5O8//VBX5dIekkKVYT6ipGURBr8Do+HrXs1t0SB2QjEzyuN+Zs1R62rDhjpNPJ/qZttS1rRzt+lCfdKkcRtDxZl2FqRXAETbggTwh0RjN1td1ONmwmwaeMLd429jvblQblR49hYqGfYNQMcwvjzt874JlVeWF8zz0nyKL9v66AbTuy2Fk= ceph-6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:13:58.192 INFO:teuthology.orchestra.run.smithi179.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:13:58.192 INFO:teuthology.orchestra.run.smithi179.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T11:13:58.193 INFO:teuthology.orchestra.run.smithi179.stdout:Adding host smithi179... 2024-03-24T11:13:58.811 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:58 smithi179 ceph-mon[29608]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:59 smithi179 ceph-mon[29608]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "addr": "172.21.15.179", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:13:59 smithi179 ceph-mon[29608]: Deploying cephadm binary to smithi179 2024-03-24T11:14:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout Added host 'smithi179' with addr '172.21.15.179' 2024-03-24T11:14:03.624 INFO:teuthology.orchestra.run.smithi179.stdout:Deploying unmanaged mon service... 2024-03-24T11:14:04.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:04 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:14:04.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:04 smithi179 ceph-mon[29608]: Added host smithi179 2024-03-24T11:14:04.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:04 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T11:14:04.891 INFO:teuthology.orchestra.run.smithi179.stdout:Deploying unmanaged mgr service... 2024-03-24T11:14:05.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:05 smithi179 ceph-mon[29608]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:05.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:05 smithi179 ceph-mon[29608]: Saving service mon spec with placement count:5 2024-03-24T11:14:05.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:05 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:14:06.485 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T11:14:06.862 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:06 smithi179 ceph-mon[29608]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:06.863 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:06 smithi179 ceph-mon[29608]: Saving service mgr spec with placement count:2 2024-03-24T11:14:06.863 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:06 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:14:08.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:07 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:14:08.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:07 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:14:08.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:07 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1649849549' entity='client.admin' 2024-03-24T11:14:09.489 INFO:teuthology.orchestra.run.smithi179.stdout:Enabling the dashboard module... 2024-03-24T11:14:10.155 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1322499615' entity='client.admin' 2024-03-24T11:14:11.116 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:10 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2031805970' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:14:11.116 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:10 smithi179 ceph-mon[29608]: from='mgr.14118 172.21.15.179:0/3263147641' entity='mgr.y' 2024-03-24T11:14:11.116 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:10 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ignoring --setuser ceph since I am not root 2024-03-24T11:14:11.116 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:10 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ignoring --setgroup ceph since I am not root 2024-03-24T11:14:11.653 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:11 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:11.378+0000 7ff7bd149200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:14:11.654 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:11 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:11.480+0000 7ff7bd149200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:14:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2031805970' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:14:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:11 smithi179 ceph-mon[29608]: mgrmap e9: y(active, since 19s) 2024-03-24T11:14:12.384 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:12.142+0000 7ff7bd149200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:14:12.384 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:12.238+0000 7ff7bd149200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:14:12.384 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:12.383+0000 7ff7bd149200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:14:12.698 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:12.480+0000 7ff7bd149200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:14:12.698 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:12.563+0000 7ff7bd149200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for the mgr to restart... 2024-03-24T11:14:12.701 INFO:teuthology.orchestra.run.smithi179.stdout:Waiting for mgr epoch 9... 2024-03-24T11:14:12.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/920329431' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:14:12.952 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:12.833+0000 7ff7bd149200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:14:12.952 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:12 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:12.945+0000 7ff7bd149200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:14:13.210 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:13 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:13.034+0000 7ff7bd149200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:14:13.210 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:13 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:13.133+0000 7ff7bd149200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:14:13.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:13 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:13.314+0000 7ff7bd149200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:14:14.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:14.275+0000 7ff7bd149200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:14:14.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:14.406+0000 7ff7bd149200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:14:15.028 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:14.742+0000 7ff7bd149200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:14:15.028 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:14.832+0000 7ff7bd149200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:14:15.028 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:15.027+0000 7ff7bd149200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:14:15.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:15.212+0000 7ff7bd149200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:14:15.756 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:15.495+0000 7ff7bd149200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:14:15.756 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:15.756+0000 7ff7bd149200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:14:16.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:15.991+0000 7ff7bd149200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:14:17.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:16 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:16.791+0000 7ff7bd149200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:14:17.675 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:17.361+0000 7ff7bd149200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:14:17.676 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:17.459+0000 7ff7bd149200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:14:17.676 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:14:17.675+0000 7ff7bd149200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:14:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: Active manager daemon y restarted 2024-03-24T11:14:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: Activating manager daemon y 2024-03-24T11:14:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:14:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: mgrmap e10: y(active, starting, since 0.00323294s) 2024-03-24T11:14:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:14:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:14:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:14:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:14:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:14:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: Manager daemon y is now available 2024-03-24T11:14:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:18 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:14:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:18 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:14:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:18 smithi179 ceph-mon[29608]: mgrmap e11: y(active, since 1.00546s) 2024-03-24T11:14:19.363 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:14:19.363 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T11:14:19.363 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:14:19.363 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:14:19.363 INFO:teuthology.orchestra.run.smithi179.stdout:mgr epoch 9 is available 2024-03-24T11:14:19.363 INFO:teuthology.orchestra.run.smithi179.stdout:Generating a dashboard self-signed certificate... 2024-03-24T11:14:20.048 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:19 smithi179 ceph-mon[29608]: [24/Mar/2024:11:14:18] ENGINE Bus STARTING 2024-03-24T11:14:20.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:19 smithi179 ceph-mon[29608]: [24/Mar/2024:11:14:18] ENGINE Serving on https://172.21.15.179:7150 2024-03-24T11:14:20.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:19 smithi179 ceph-mon[29608]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:14:20.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:19 smithi179 ceph-mon[29608]: [24/Mar/2024:11:14:18] ENGINE Serving on http://172.21.15.179:8765 2024-03-24T11:14:20.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:19 smithi179 ceph-mon[29608]: [24/Mar/2024:11:14:18] ENGINE Bus STARTED 2024-03-24T11:14:20.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:19 smithi179 ceph-mon[29608]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:14:20.986 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:20 smithi179 ceph-mon[29608]: mgrmap e12: y(active, since 2s) 2024-03-24T11:14:20.987 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:20.987 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:20.987 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:20.989 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T11:14:20.989 INFO:teuthology.orchestra.run.smithi179.stdout:Creating initial admin user... 2024-03-24T11:14:22.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:21 smithi179 ceph-mon[29608]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:22.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:22.977 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$FAV60Bt2uy6UWlPyl7JBV.7U6YWnMGyh57RXUnqJtEcwlyf8aPb3m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711278861, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:14:22.977 INFO:teuthology.orchestra.run.smithi179.stdout:Fetching dashboard port number... 2024-03-24T11:14:23.076 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:22 smithi179 ceph-mon[29608]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:23.077 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:22 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:24.704 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T11:14:24.713 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:14:24.713 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout disabled 2024-03-24T11:14:24.721 INFO:teuthology.orchestra.run.smithi179.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:14:24.721 INFO:teuthology.orchestra.run.smithi179.stdout:systemctl: stdout inactive 2024-03-24T11:14:24.721 INFO:teuthology.orchestra.run.smithi179.stdout:firewalld.service is not enabled 2024-03-24T11:14:24.721 INFO:teuthology.orchestra.run.smithi179.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:14:24.723 INFO:teuthology.orchestra.run.smithi179.stdout:Ceph Dashboard is now available at: 2024-03-24T11:14:24.723 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:24.723 INFO:teuthology.orchestra.run.smithi179.stdout: URL: https://smithi179.front.sepia.ceph.com:8443/ 2024-03-24T11:14:24.723 INFO:teuthology.orchestra.run.smithi179.stdout: User: admin 2024-03-24T11:14:24.723 INFO:teuthology.orchestra.run.smithi179.stdout: Password: a66jt1dkwi 2024-03-24T11:14:24.723 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:24.723 INFO:teuthology.orchestra.run.smithi179.stdout:Saving cluster configuration to /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config directory 2024-03-24T11:14:24.724 INFO:teuthology.orchestra.run.smithi179.stdout:Enabling autotune for osd_memory_target 2024-03-24T11:14:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:24 smithi179 ceph-mon[29608]: mgrmap e13: y(active, since 6s) 2024-03-24T11:14:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:24 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1080395674' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:14:27.345 INFO:teuthology.orchestra.run.smithi179.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T11:14:27.345 INFO:teuthology.orchestra.run.smithi179.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T11:14:27.345 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.345 INFO:teuthology.orchestra.run.smithi179.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: ceph telemetry on 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout:For more information see: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.346 INFO:teuthology.orchestra.run.smithi179.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T11:14:27.347 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:27.347 INFO:teuthology.orchestra.run.smithi179.stdout:Bootstrap complete. 2024-03-24T11:14:27.390 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:14:27.391 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:14:27.391 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:14:27.407 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:14:27.407 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:14:27.407 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:14:27.463 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:14:27.463 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:14:27.463 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T11:14:27.530 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:14:27.531 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:14:27.531 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:14:27.585 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:14:27.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/f2JVCdcpDrSHkG9kt69CP7jDodmLHL21YJI8dX30lRbwWeDHW+nViQbkF2Ivc8fc5yUe8XfGSkUuZKHJiBVmsIwUHwsN+jPXFpoQNldpmIsnf7YO/+iaFNubA593TwuOQfxjoZfMxOLYP3brrdLu2VoDeg9zfaMl/sUll+9p1TggB2rgqB/ULea+UwhQEfm+1mP2PHAU4hFb+hzk2u6GwZL/1PpIW/bstzPKd49rNQAANNBZ8FkoAqC6hjJX/BqBMgW3Olepk6YWKSsUN7EDkPrG3YioBev2Cr1BV1Dn5TUiWmIB6jccNPmAqLr/tB2vxTx/COoO4porxP5O8//VBX5dIekkKVYT6ipGURBr8Do+HrXs1t0SB2QjEzyuN+Zs1R62rDhjpNPJ/qZttS1rRzt+lCfdKkcRtDxZl2FqRXAETbggTwh0RjN1td1ONmwmwaeMLd429jvblQblR49hYqGfYNQMcwvjzt874JlVeWF8zz0nyKL9v66AbTuy2Fk= ceph-6430538c-e9cf-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:14:27.678 INFO:teuthology.orchestra.run.smithi179.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/f2JVCdcpDrSHkG9kt69CP7jDodmLHL21YJI8dX30lRbwWeDHW+nViQbkF2Ivc8fc5yUe8XfGSkUuZKHJiBVmsIwUHwsN+jPXFpoQNldpmIsnf7YO/+iaFNubA593TwuOQfxjoZfMxOLYP3brrdLu2VoDeg9zfaMl/sUll+9p1TggB2rgqB/ULea+UwhQEfm+1mP2PHAU4hFb+hzk2u6GwZL/1PpIW/bstzPKd49rNQAANNBZ8FkoAqC6hjJX/BqBMgW3Olepk6YWKSsUN7EDkPrG3YioBev2Cr1BV1Dn5TUiWmIB6jccNPmAqLr/tB2vxTx/COoO4porxP5O8//VBX5dIekkKVYT6ipGURBr8Do+HrXs1t0SB2QjEzyuN+Zs1R62rDhjpNPJ/qZttS1rRzt+lCfdKkcRtDxZl2FqRXAETbggTwh0RjN1td1ONmwmwaeMLd429jvblQblR49hYqGfYNQMcwvjzt874JlVeWF8zz0nyKL9v66AbTuy2Fk= ceph-6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:14:27.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/f2JVCdcpDrSHkG9kt69CP7jDodmLHL21YJI8dX30lRbwWeDHW+nViQbkF2Ivc8fc5yUe8XfGSkUuZKHJiBVmsIwUHwsN+jPXFpoQNldpmIsnf7YO/+iaFNubA593TwuOQfxjoZfMxOLYP3brrdLu2VoDeg9zfaMl/sUll+9p1TggB2rgqB/ULea+UwhQEfm+1mP2PHAU4hFb+hzk2u6GwZL/1PpIW/bstzPKd49rNQAANNBZ8FkoAqC6hjJX/BqBMgW3Olepk6YWKSsUN7EDkPrG3YioBev2Cr1BV1Dn5TUiWmIB6jccNPmAqLr/tB2vxTx/COoO4porxP5O8//VBX5dIekkKVYT6ipGURBr8Do+HrXs1t0SB2QjEzyuN+Zs1R62rDhjpNPJ/qZttS1rRzt+lCfdKkcRtDxZl2FqRXAETbggTwh0RjN1td1ONmwmwaeMLd429jvblQblR49hYqGfYNQMcwvjzt874JlVeWF8zz0nyKL9v66AbTuy2Fk= ceph-6430538c-e9cf-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:14:27.740 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/f2JVCdcpDrSHkG9kt69CP7jDodmLHL21YJI8dX30lRbwWeDHW+nViQbkF2Ivc8fc5yUe8XfGSkUuZKHJiBVmsIwUHwsN+jPXFpoQNldpmIsnf7YO/+iaFNubA593TwuOQfxjoZfMxOLYP3brrdLu2VoDeg9zfaMl/sUll+9p1TggB2rgqB/ULea+UwhQEfm+1mP2PHAU4hFb+hzk2u6GwZL/1PpIW/bstzPKd49rNQAANNBZ8FkoAqC6hjJX/BqBMgW3Olepk6YWKSsUN7EDkPrG3YioBev2Cr1BV1Dn5TUiWmIB6jccNPmAqLr/tB2vxTx/COoO4porxP5O8//VBX5dIekkKVYT6ipGURBr8Do+HrXs1t0SB2QjEzyuN+Zs1R62rDhjpNPJ/qZttS1rRzt+lCfdKkcRtDxZl2FqRXAETbggTwh0RjN1td1ONmwmwaeMLd429jvblQblR49hYqGfYNQMcwvjzt874JlVeWF8zz0nyKL9v66AbTuy2Fk= ceph-6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:14:27.755 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:14:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:27 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2848913083' entity='client.admin' 2024-03-24T11:14:27.998 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:14:30.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:30 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:30.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:30 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:30.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:30 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:30.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:30 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:30.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:30 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:30.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:30 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:31.136 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:14:31.136 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:14:31.372 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:14:31.779 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:31 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/273576422' entity='client.admin' 2024-03-24T11:14:31.780 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:31 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:31.780 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:31 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:33.832 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-03-24T11:14:33.832 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:14:33.833 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:14:33.851 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:14:33.851 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:14:33.907 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-03-24T11:14:33.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch host add smithi181 2024-03-24T11:14:34.139 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:14:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-24T11:14:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:14:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:14:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:14:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:36 smithi179 ceph-mon[29608]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:36 smithi179 ceph-mon[29608]: Deploying cephadm binary to smithi181 2024-03-24T11:14:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:38 smithi179 ceph-mon[29608]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:40.150 INFO:teuthology.orchestra.run.smithi179.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-03-24T11:14:40.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:40 smithi179 ceph-mon[29608]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:40.803 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:40.803 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:40.803 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:40.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:14:41.062 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:14:41.850 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:41 smithi179 ceph-mon[29608]: Added host smithi181 2024-03-24T11:14:41.850 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:42.856 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:14:42.856 INFO:teuthology.orchestra.run.smithi179.stdout:[{"addr": "172.21.15.179", "hostname": "smithi179", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-03-24T11:14:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:42 smithi179 ceph-mon[29608]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:43.419 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:14:43.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:14:43.650 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:14:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:43 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:43 smithi179 ceph-mon[29608]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:14:45.056 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:44 smithi179 ceph-mon[29608]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:45.716 INFO:teuthology.orchestra.run.smithi179.stderr:adjusted tunables profile to default 2024-03-24T11:14:46.057 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:45 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2733220323' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:14:46.352 INFO:tasks.cephadm:Adding mon.a on smithi179 2024-03-24T11:14:46.352 INFO:tasks.cephadm:Adding mon.c on smithi179 2024-03-24T11:14:46.352 INFO:tasks.cephadm:Adding mon.b on smithi181 2024-03-24T11:14:46.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi179:172.21.15.179=a;smithi179:[v2:172.21.15.179:3301,v1:172.21.15.179:6790]=c;smithi181:172.21.15.181=b' 2024-03-24T11:14:46.588 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:14:47.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:47.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2733220323' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:47 smithi179 ceph-mon[29608]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:14:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:47 smithi179 ceph-mon[29608]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:14:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:47 smithi179 ceph-mon[29608]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:14:48.319 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-03-24T11:14:48.950 DEBUG:teuthology.orchestra.run.smithi179:mon.c> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T11:14:48.953 DEBUG:teuthology.orchestra.run.smithi181:mon.b> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T11:14:48.955 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:14:48.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:14:48.979 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:48.979 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:48.979 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:48.979 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:48.980 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:48.980 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:14:48.980 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:14:48.980 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:49.635 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:14:50.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:49 smithi179 ceph-mon[29608]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi179:172.21.15.179=a;smithi179:[v2:172.21.15.179:3301,v1:172.21.15.179:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:50.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:49 smithi179 ceph-mon[29608]: Saving service mon spec with placement smithi179:172.21.15.179=a;smithi179:[v2:172.21.15.179:3301,v1:172.21.15.179:6790]=c;smithi181:172.21.15.181=b;count:3 2024-03-24T11:14:50.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:49 smithi179 ceph-mon[29608]: Deploying daemon mon.b on smithi181 2024-03-24T11:14:51.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:50 smithi179 ceph-mon[29608]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:51.792 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:14:51.792 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","modified":"2024-03-24T11:13:15.735830Z","created":"2024-03-24T11:13:15.735830Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:14:51.794 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-03-24T11:14:53.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:52 smithi179 ceph-mon[29608]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:53.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:52 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/319797036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:14:53.387 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:14:53.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:14:53.724 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:14:55.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:14:54 smithi179 ceph-mon[29608]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:14:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:14:55 smithi181 systemd[1]: Started Ceph mon.b for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:15:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:15:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: mon.a calling monitor election 2024-03-24T11:15:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: mon.b calling monitor election 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: monmap e2: 2 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: fsmap 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:15:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: mgrmap e13: y(active, since 42s) 2024-03-24T11:15:00.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: overall HEALTH_OK 2024-03-24T11:15:00.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:00.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:15:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: mon.a calling monitor election 2024-03-24T11:15:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: mon.b calling monitor election 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: monmap e2: 2 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: fsmap 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: mgrmap e13: y(active, since 42s) 2024-03-24T11:15:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: overall HEALTH_OK 2024-03-24T11:15:00.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:00.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:01 smithi179 ceph-mon[29608]: Deploying daemon mon.c on smithi179 2024-03-24T11:15:01.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:01.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:01.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:01.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:01.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:01 smithi181 ceph-mon[33998]: Deploying daemon mon.c on smithi179 2024-03-24T11:15:01.742 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:01.743 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:15:01.743 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","modified":"2024-03-24T11:14:55.339340Z","created":"2024-03-24T11:13:15.735830Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:15:01.743 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-03-24T11:15:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:02 smithi181 ceph-mon[33998]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:02 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/2769239379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:15:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:02 smithi179 ceph-mon[29608]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:02 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/2769239379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:15:03.298 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:15:03.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:15:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 systemd[1]: Starting Ceph mon.c for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:15:03.541 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:15:03.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 podman[37888]: 2024-03-24 11:15:03.576747923 +0000 UTC m=+0.101808426 container create 13ecfd493a2dea82f8d07c9821cb7eeaa69852acf9d1e5709c78d25e6554915f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-24T11:15:03.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 podman[37888]: 2024-03-24 11:15:03.492044331 +0000 UTC m=+0.017104841 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:15:03.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 podman[37888]: 2024-03-24 11:15:03.813318327 +0000 UTC m=+0.338378840 container init 13ecfd493a2dea82f8d07c9821cb7eeaa69852acf9d1e5709c78d25e6554915f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T11:15:03.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 podman[37888]: 2024-03-24 11:15:03.816558618 +0000 UTC m=+0.341619123 container start 13ecfd493a2dea82f8d07c9821cb7eeaa69852acf9d1e5709c78d25e6554915f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 2 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pidfile_write: ignore empty --pid-file 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: load: jerasure load: lrc 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Git sha 0 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: DB SUMMARY 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: DB Session ID: QI8B2FFP7MD73UP95OCN 2024-03-24T11:15:04.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.env: 0x55d76f2e4ae0 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:15:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.info_log: 0x55d7712578c0 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.statistics: (nil) 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.use_fsync: 0 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:15:04.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.db_log_dir: 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.wal_dir: 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:15:04.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.write_buffer_manager: 0x55d77122b400 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.unordered_write: 0 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:15:04.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.row_cache: None 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.wal_filter: None 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.wal_compression: 0 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:15:04.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:15:04.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:15:04.218 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:15:04.218 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:15:04.218 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:15:04.218 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_open_files: -1 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:15:04.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Compression algorithms supported: 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kZSTD supported: 0 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:15:04.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.merge_operator: 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_filter: None 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d7712579c0) 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:15:04.221 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_type: 0 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_index_type: 0 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_shortening: 1 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: checksum: 4 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: no_block_cache: 0 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache: 0x55d77125b090 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_options: 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: capacity : 536870912 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_shard_bits : 4 2024-03-24T11:15:04.222 INFO:journalctl@ceph.mon.c.smithi179.stdout: strict_capacity_limit : 0 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_compressed: (nil) 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: persistent_cache: (nil) 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size: 4096 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size_deviation: 10 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_restart_interval: 16 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_block_restart_interval: 1 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: metadata_block_size: 4096 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: partition_filters: 0 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: use_delta_encoding: 1 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: filter_policy: bloomfilter 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: whole_key_filtering: 1 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: verify_compression: 0 2024-03-24T11:15:04.223 INFO:journalctl@ceph.mon.c.smithi179.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout: format_version: 5 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_index_compression: 1 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_align: 0 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout: max_auto_readahead_size: 262144 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout: prepopulate_block_cache: 0 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression: NoCompression 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:15:04.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.num_levels: 7 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:15:04.225 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:15:04.226 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:15:04.227 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:15:04.228 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:15:04.229 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.ttl: 2592000 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:15:04.230 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3288df45-b2d1-40da-a104-c5b927169b6d 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278903846151, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278903846556, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711278903, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3288df45-b2d1-40da-a104-c5b927169b6d", "db_session_id": "QI8B2FFP7MD73UP95OCN", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278903846636, "job": 1, "event": "recovery_finished"} 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:15:04.231 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d77134c000 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: DB pointer 0x55d771336000 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** DB Stats ** 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-03-24T11:15:04.232 INFO:journalctl@ceph.mon.c.smithi179.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:15:04.233 INFO:journalctl@ceph.mon.c.smithi179.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache BinnedLRUCache@0x55d77125b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 7e-06 secs_since: 0 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:15:04.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: using public_addrv [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: starting mon.c rank -1 at public addrs [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] at bind addrs [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(???) e0 preinit fsid 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout: e1 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout: legacy client fscid: -1 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout: No filesystems configured 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:15:04.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2733220323' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2733220323' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.236 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:15:04.237 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi179:172.21.15.179=a;smithi179:[v2:172.21.15.179:3301,v1:172.21.15.179:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: Saving service mon spec with placement smithi179:172.21.15.179=a;smithi179:[v2:172.21.15.179:3301,v1:172.21.15.179:6790]=c;smithi181:172.21.15.181=b;count:3 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: Deploying daemon mon.b on smithi181 2024-03-24T11:15:04.238 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/319797036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.a calling monitor election 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.b calling monitor election 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:04.239 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: monmap e2: 2 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: fsmap 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mgrmap e13: y(active, since 42s) 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: overall HEALTH_OK 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: Deploying daemon mon.c on smithi179 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:04.240 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/2769239379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:15:04.241 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 ceph-mon[37907]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T11:15:04.241 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 bash[37888]: 13ecfd493a2dea82f8d07c9821cb7eeaa69852acf9d1e5709c78d25e6554915f 2024-03-24T11:15:04.241 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:03 smithi179 systemd[1]: Started Ceph mon.c for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:15:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:15:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: mon.a calling monitor election 2024-03-24T11:15:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: mon.b calling monitor election 2024-03-24T11:15:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: mon.c calling monitor election 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: monmap e3: 3 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: fsmap 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: mgrmap e13: y(active, since 51s) 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: overall HEALTH_OK 2024-03-24T11:15:09.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:15:09.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: mon.a calling monitor election 2024-03-24T11:15:09.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: mon.b calling monitor election 2024-03-24T11:15:09.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:09.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: mon.c calling monitor election 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: monmap e3: 3 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: fsmap 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: mgrmap e13: y(active, since 51s) 2024-03-24T11:15:09.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: overall HEALTH_OK 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: mon.a calling monitor election 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: mon.b calling monitor election 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: mon.c calling monitor election 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: monmap e3: 3 mons at {a=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: fsmap 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: mgrmap e13: y(active, since 51s) 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: overall HEALTH_OK 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:09.670 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-03-24T11:15:09.671 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","modified":"2024-03-24T11:15:03.865852Z","created":"2024-03-24T11:13:15.735830Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3301","nonce":0},{"type":"v1","addr":"172.21.15.179:6790","nonce":0}]},"addr":"172.21.15.179:6790/0","public_addr":"172.21.15.179:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-24T11:15:09.675 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-03-24T11:15:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3041285051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:15:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[37907]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:10.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:10.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:10.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3041285051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:15:10.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[29608]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:10.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:09 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:10.226 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:09 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:10.226 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3041285051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:15:10.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:09 smithi181 ceph-mon[33998]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:10.227 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:09 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:15:10.468 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:15:10.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:15:10.701 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-24T11:15:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:11.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:11.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:11.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[29608]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:15:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[29608]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:15:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[29608]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:15:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[29608]: Reconfiguring daemon mon.a on smithi179 2024-03-24T11:15:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[29608]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:12.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[37907]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:15:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[37907]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:15:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[37907]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:15:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[37907]: Reconfiguring daemon mon.a on smithi179 2024-03-24T11:15:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:12 smithi179 ceph-mon[37907]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:12 smithi181 ceph-mon[33998]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:15:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:12 smithi181 ceph-mon[33998]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:15:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:12 smithi181 ceph-mon[33998]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T11:15:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:12 smithi181 ceph-mon[33998]: Reconfiguring daemon mon.a on smithi179 2024-03-24T11:15:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:12 smithi181 ceph-mon[33998]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:13.159 INFO:teuthology.orchestra.run.smithi179.stdout:# minimal ceph.conf for 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:15:13.160 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2024-03-24T11:15:13.160 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:15:13.160 INFO:teuthology.orchestra.run.smithi179.stdout: mon_host = [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] 2024-03-24T11:15:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2058529902' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2058529902' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:13.996 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:15:13.996 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:15:13.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:15:14.026 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:15:14.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:15:14.093 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:15:14.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:15:14.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:14.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:13 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2058529902' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:14.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:14.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:14.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:14.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:14.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:14.129 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:15:14.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:15:14.199 INFO:tasks.cephadm:Adding mgr.y on smithi179 2024-03-24T11:15:14.199 INFO:tasks.cephadm:Adding mgr.x on smithi181 2024-03-24T11:15:14.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi179=y;smithi181=x' 2024-03-24T11:15:14.471 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:15:14.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:14 smithi181 ceph-mon[33998]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:15:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:14 smithi181 ceph-mon[33998]: Reconfiguring daemon mon.c on smithi179 2024-03-24T11:15:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:14 smithi181 ceph-mon[33998]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:15.102 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:14 smithi179 ceph-mon[29608]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:15:15.102 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:14 smithi179 ceph-mon[29608]: Reconfiguring daemon mon.c on smithi179 2024-03-24T11:15:15.102 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:14 smithi179 ceph-mon[29608]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:15.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:14 smithi179 ceph-mon[37907]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:15:15.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:14 smithi179 ceph-mon[37907]: Reconfiguring daemon mon.c on smithi179 2024-03-24T11:15:15.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:14 smithi179 ceph-mon[37907]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:16.206 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: Reconfiguring daemon mon.b on smithi181 2024-03-24T11:15:16.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:16 smithi181 ceph-mon[33998]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:16.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:16.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:16.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:15:16.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:16.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:16.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:16.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: Reconfiguring daemon mon.b on smithi181 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[29608]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: Reconfiguring daemon mon.b on smithi181 2024-03-24T11:15:16.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:16 smithi179 ceph-mon[37907]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:17.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi179=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: Saving service mgr spec with placement smithi179=y;smithi181=x;count:2 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:15:17.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi179=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: Saving service mgr spec with placement smithi179=y;smithi181=x;count:2 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:15:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:15:17.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:17.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi179=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:17.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: Saving service mgr spec with placement smithi179=y;smithi181=x;count:2 2024-03-24T11:15:17.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.474 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:15:17.475 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:17.476 DEBUG:teuthology.orchestra.run.smithi181:mgr.x> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T11:15:17.478 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:15:17.478 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:15:17.479 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:15:17.499 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:15:17.499 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-03-24T11:15:17.557 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:15:17.557 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:17.557 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T11:15:17.557 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:17.557 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:17.557 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-24 11:15:03.314487093 +0000 2024-03-24T11:15:17.558 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-24 11:09:07.927118045 +0000 2024-03-24T11:15:17.558 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-24 11:09:07.927118045 +0000 2024-03-24T11:15:17.558 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-24 11:09:07.927118045 +0000 2024-03-24T11:15:17.558 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:15:17.628 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-24T11:15:17.628 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-24T11:15:17.628 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000228972 s, 2.2 MB/s 2024-03-24T11:15:17.629 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:15:17.687 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-24 11:15:03.314487093 +0000 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-24 11:09:08.257110852 +0000 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-24 11:09:08.257110852 +0000 2024-03-24T11:15:17.744 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-24 11:09:08.257110852 +0000 2024-03-24T11:15:17.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:15:17.813 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-24T11:15:17.813 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-24T11:15:17.813 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000226827 s, 2.3 MB/s 2024-03-24T11:15:17.814 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:15:17.871 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-03-24T11:15:17.927 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:15:17.927 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:17.927 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T11:15:17.928 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:17.928 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:17.928 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-24 11:15:03.315487072 +0000 2024-03-24T11:15:17.928 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-24 11:09:08.603103310 +0000 2024-03-24T11:15:17.928 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-24 11:09:08.603103310 +0000 2024-03-24T11:15:17.928 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-24 11:09:08.603103310 +0000 2024-03-24T11:15:17.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:15:17.994 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-24T11:15:17.994 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-24T11:15:17.995 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000209935 s, 2.4 MB/s 2024-03-24T11:15:17.996 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:15:18.053 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-03-24T11:15:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:15:18.109 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:18.109 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T11:15:18.109 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:18.109 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:18.109 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-24 11:15:03.315487072 +0000 2024-03-24T11:15:18.109 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-24 11:09:08.947095812 +0000 2024-03-24T11:15:18.110 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-24 11:09:08.947095812 +0000 2024-03-24T11:15:18.110 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-24 11:09:08.947095812 +0000 2024-03-24T11:15:18.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:15:18.175 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-24T11:15:18.175 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-24T11:15:18.175 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.00018536 s, 2.8 MB/s 2024-03-24T11:15:18.176 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:15:18.233 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:15:18.233 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:15:18.249 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:15:18.249 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-24 11:14:53.643537651 +0000 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-24 11:09:07.940355779 +0000 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-24 11:09:07.940355779 +0000 2024-03-24T11:15:18.304 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-03-24 11:09:07.940355779 +0000 2024-03-24T11:15:18.305 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:15:18.370 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-24T11:15:18.370 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-24T11:15:18.370 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000205668 s, 2.5 MB/s 2024-03-24T11:15:18.371 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:15:18.426 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-03-24T11:15:18.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:18 smithi179 ceph-mon[29608]: Deploying daemon mgr.x on smithi181 2024-03-24T11:15:18.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:18 smithi179 ceph-mon[29608]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:18 smithi179 ceph-mon[37907]: Deploying daemon mgr.x on smithi181 2024-03-24T11:15:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:18 smithi179 ceph-mon[37907]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:18.470 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:18 smithi181 ceph-mon[33998]: Deploying daemon mgr.x on smithi181 2024-03-24T11:15:18.470 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:18 smithi181 ceph-mon[33998]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-24 11:14:53.644537633 +0000 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-24 11:09:08.294349328 +0000 2024-03-24T11:15:18.482 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-24 11:09:08.294349328 +0000 2024-03-24T11:15:18.483 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-03-24 11:09:08.294349328 +0000 2024-03-24T11:15:18.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:15:18.548 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-24T11:15:18.548 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-24T11:15:18.548 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000187198 s, 2.7 MB/s 2024-03-24T11:15:18.549 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:15:18.610 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-03-24T11:15:18.665 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-24 11:14:53.644537633 +0000 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-24 11:09:08.643342968 +0000 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-24 11:09:08.643342968 +0000 2024-03-24T11:15:18.666 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-03-24 11:09:08.643342968 +0000 2024-03-24T11:15:18.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:15:18.733 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-24T11:15:18.733 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-24T11:15:18.733 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000216421 s, 2.4 MB/s 2024-03-24T11:15:18.734 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:15:18.792 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-03-24T11:15:18.847 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:15:18.847 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:15:18.847 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T11:15:18.847 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:15:18.847 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:15:18.847 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-03-24 11:15:18.672086030 +0000 2024-03-24T11:15:18.848 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-03-24 11:09:08.979336846 +0000 2024-03-24T11:15:18.848 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-03-24 11:09:08.979336846 +0000 2024-03-24T11:15:18.848 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-03-24 11:09:08.979336846 +0000 2024-03-24T11:15:18.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:15:18.917 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-03-24T11:15:18.917 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-03-24T11:15:18.917 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000183726 s, 2.8 MB/s 2024-03-24T11:15:18.918 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:15:18.973 INFO:tasks.cephadm:Deploying osd.0 on smithi179 with /dev/vg_nvme/lv_4... 2024-03-24T11:15:18.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:15:19.219 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:15:19.402 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:19 smithi181 systemd[1]: Started Ceph mgr.x for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:15:20.151 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:19.855+0000 7fd95fc08200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:15:20.152 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:19.955+0000 7fd95fc08200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:15:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[29608]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:20 smithi179 ceph-mon[37907]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:20.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.562 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.563 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:20.563 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:20.563 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-mon[33998]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:20.824 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:20.634+0000 7fd95fc08200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:15:20.824 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:20.731+0000 7fd95fc08200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:15:21.082 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:20.893+0000 7fd95fc08200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:15:21.082 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:20.995+0000 7fd95fc08200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:15:21.082 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:21.081+0000 7fd95fc08200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:15:21.247 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:15:21.247 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:15:21.247 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-24T11:15:21.247 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-24T11:15:21.247 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114516 s, 916 MB/s 2024-03-24T11:15:21.248 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-24T11:15:21.648 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:21.352+0000 7fd95fc08200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:15:21.648 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:21.465+0000 7fd95fc08200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:15:21.648 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:21.552+0000 7fd95fc08200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:15:21.648 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:21.648+0000 7fd95fc08200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:15:21.902 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:21.830+0000 7fd95fc08200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:15:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:15:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:15:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:21 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:15:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:15:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:21 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:22.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_4 2024-03-24T11:15:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[29608]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:15:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[29608]: Reconfiguring daemon mgr.y on smithi179 2024-03-24T11:15:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[29608]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[37907]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:15:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[37907]: Reconfiguring daemon mgr.y on smithi179 2024-03-24T11:15:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[37907]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:22 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:23.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:22 smithi181 ceph-mon[33998]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:15:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:22 smithi181 ceph-mon[33998]: Reconfiguring daemon mgr.y on smithi179 2024-03-24T11:15:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:22 smithi181 ceph-mon[33998]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:22 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:23.152 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:22 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:22.799+0000 7fd95fc08200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:15:23.152 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:22 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:22.929+0000 7fd95fc08200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:15:23.229 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:15:23.546 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:23.263+0000 7fd95fc08200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:15:23.547 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:23.352+0000 7fd95fc08200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:15:23.547 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:23.545+0000 7fd95fc08200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:15:23.901 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:23.729+0000 7fd95fc08200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:15:24.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:24.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-mon[33998]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:24.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:24.278 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:24.278 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.278 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:24.011+0000 7fd95fc08200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:15:24.278 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:24.275+0000 7fd95fc08200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:15:24.438 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.438 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.438 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:24.438 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[29608]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[37907]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:24.439 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:24.651 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:24.512+0000 7fd95fc08200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:15:25.643 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:25 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:25.310+0000 7fd95fc08200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:15:25.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:25 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:15:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:25 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:15:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:25 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:25.902 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:25 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:25.881+0000 7fd95fc08200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:15:25.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:25 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:15:25.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:25 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:15:25.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:25 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:25.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:25 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:15:25.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:25 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:15:25.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:25 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:26.402 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:25 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:25.978+0000 7fd95fc08200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:15:26.402 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:15:26.192+0000 7fd95fc08200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:15:26.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-mon[33998]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-mon[33998]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-mon[33998]: Standby manager daemon x started 2024-03-24T11:15:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:15:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:15:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:15:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:26 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[29608]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[29608]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[29608]: Standby manager daemon x started 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[37907]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[37907]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[37907]: Standby manager daemon x started 2024-03-24T11:15:26.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:15:26.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:15:26.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:15:26.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:26 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/171659055' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:15:27.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:27 smithi181 ceph-mon[33998]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-24T11:15:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:27 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:15:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:27 smithi179 ceph-mon[29608]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-24T11:15:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:27 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:15:27.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:27 smithi179 ceph-mon[37907]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-24T11:15:27.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:27 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:15:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[29608]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1271629540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]: dispatch 2024-03-24T11:15:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]: dispatch 2024-03-24T11:15:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]': finished 2024-03-24T11:15:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[29608]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:15:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2275502393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[37907]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1271629540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]: dispatch 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]: dispatch 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]': finished 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[37907]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2275502393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:29.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:28 smithi181 ceph-mon[33998]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1271629540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]: dispatch 2024-03-24T11:15:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:28 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]: dispatch 2024-03-24T11:15:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:28 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b"}]': finished 2024-03-24T11:15:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:28 smithi181 ceph-mon[33998]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:15:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:28 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2275502393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:30 smithi179 ceph-mon[29608]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:30.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:30 smithi179 ceph-mon[37907]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:31.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:30 smithi181 ceph-mon[33998]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:33.016 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:32 smithi179 ceph-mon[29608]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:33.016 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:32 smithi179 ceph-mon[37907]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:32 smithi181 ceph-mon[33998]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:34.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:34 smithi179 ceph-mon[29608]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:34.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:15:34.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:34.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:34 smithi179 ceph-mon[37907]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:34.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:34 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:15:34.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:34 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:34.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:34 smithi181 ceph-mon[33998]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:34 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:15:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:34 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:35.864 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:35 smithi179 ceph-mon[29608]: Deploying daemon osd.0 on smithi179 2024-03-24T11:15:35.864 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:35 smithi179 ceph-mon[37907]: Deploying daemon osd.0 on smithi179 2024-03-24T11:15:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:35 smithi181 ceph-mon[33998]: Deploying daemon osd.0 on smithi179 2024-03-24T11:15:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:36 smithi179 ceph-mon[29608]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:36.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:36 smithi179 ceph-mon[37907]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:37.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:36 smithi181 ceph-mon[33998]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:38.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:38 smithi179 ceph-mon[29608]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:38.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:38 smithi179 ceph-mon[37907]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:39.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:38 smithi181 ceph-mon[33998]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[29608]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:40.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:40.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:40.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[37907]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:41.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:40 smithi181 ceph-mon[33998]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:41.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:41.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:41.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:42 smithi181 ceph-mon[33998]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:42 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:42 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:42 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:42 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:42 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[37907]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[29608]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.191 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.192 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:43.192 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:43.192 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:42 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:43.486 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 0 on host 'smithi179' 2024-03-24T11:15:44.096 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[29608]: from='osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:15:44.096 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[29608]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[37907]: from='osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[37907]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:44.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:43 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:44.126 DEBUG:teuthology.orchestra.run.smithi179:osd.0> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:15:44.128 INFO:tasks.cephadm:Deploying osd.1 on smithi179 with /dev/vg_nvme/lv_3... 2024-03-24T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:15:44.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:43 smithi181 ceph-mon[33998]: from='osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:15:44.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:43 smithi181 ceph-mon[33998]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:15:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:43 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:43 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:43 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:45.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[37907]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:45.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[37907]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[37907]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[37907]: from='osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[37907]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[29608]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[29608]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[29608]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[29608]: from='osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:15:45.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:44 smithi179 ceph-mon[29608]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:15:45.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:44 smithi181 ceph-mon[33998]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:44 smithi181 ceph-mon[33998]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:15:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:44 smithi181 ceph-mon[33998]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:15:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:44 smithi181 ceph-mon[33998]: from='osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:15:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:44 smithi181 ceph-mon[33998]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:15:45.204 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:15:45.432 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0[41905]: 2024-03-24T11:15:45.238+0000 7f88a2c19700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T11:15:45.432 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0[41905]: 2024-03-24T11:15:45.245+0000 7f889a203700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:15:45.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[29608]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:15:45.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[29608]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:15:45.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[29608]: from='osd.0 ' entity='osd.0' 2024-03-24T11:15:45.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[37907]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:15:45.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[37907]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:15:45.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:45.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[37907]: from='osd.0 ' entity='osd.0' 2024-03-24T11:15:45.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:45 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:45 smithi181 ceph-mon[33998]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:15:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:45 smithi181 ceph-mon[33998]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:15:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:45 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:45 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:45 smithi181 ceph-mon[33998]: from='osd.0 ' entity='osd.0' 2024-03-24T11:15:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:45 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:15:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:15:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[29608]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[29608]: osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377] boot 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[29608]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[37907]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[37907]: osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377] boot 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[37907]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:15:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:46 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:47.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:46 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:15:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:46 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:15:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:46 smithi181 ceph-mon[33998]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:15:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:46 smithi181 ceph-mon[33998]: osd.0 [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377] boot 2024-03-24T11:15:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:46 smithi181 ceph-mon[33998]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:15:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:46 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:15:47.817 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:15:47.817 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:15:47.818 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-24T11:15:47.818 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-24T11:15:47.818 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125905 s, 833 MB/s 2024-03-24T11:15:47.818 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-24T11:15:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:15:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi179 2024-03-24T11:15:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi179 to 16064M 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[29608]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi179 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi179 to 16064M 2024-03-24T11:15:48.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:48.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:48.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:48 smithi179 ceph-mon[37907]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:48.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_3 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi179 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi179 to 16064M 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:15:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:48 smithi181 ceph-mon[33998]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:48.762 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:15:50.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[29608]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:50.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:15:50.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:15:50.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:50.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[37907]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:50.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:15:50.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:15:50.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:50 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:51.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:50 smithi181 ceph-mon[33998]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:50 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:15:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:50 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:15:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:50 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:52.125 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:51 smithi179 ceph-mon[29608]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:52.125 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:51 smithi179 ceph-mon[37907]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:52.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:51 smithi181 ceph-mon[33998]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:15:53.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:52 smithi181 ceph-mon[33998]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:53.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:52 smithi179 ceph-mon[29608]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:52 smithi179 ceph-mon[37907]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:54.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1836420487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]: dispatch 2024-03-24T11:15:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:53 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]: dispatch 2024-03-24T11:15:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:53 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]': finished 2024-03-24T11:15:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:53 smithi181 ceph-mon[33998]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:15:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:53 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:15:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1836420487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]: dispatch 2024-03-24T11:15:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]: dispatch 2024-03-24T11:15:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]': finished 2024-03-24T11:15:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[29608]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:15:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:15:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1836420487' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]: dispatch 2024-03-24T11:15:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]: dispatch 2024-03-24T11:15:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667"}]': finished 2024-03-24T11:15:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[37907]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:15:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:53 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:15:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:54 smithi181 ceph-mon[33998]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/575660693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:55.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:54 smithi179 ceph-mon[29608]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:55.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/575660693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:54 smithi179 ceph-mon[37907]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/575660693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:56.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:56 smithi179 ceph-mon[37907]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:56.928 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:56 smithi179 ceph-mon[29608]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:57.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:56 smithi181 ceph-mon[33998]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:58.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:15:58 smithi179 ceph-mon[29608]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:58.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:15:58 smithi179 ceph-mon[37907]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:15:59.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:15:58 smithi181 ceph-mon[33998]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:00.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:00 smithi179 ceph-mon[29608]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:00.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:16:00.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:00 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:00.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:00 smithi179 ceph-mon[37907]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:00.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:00 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:16:00.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:00 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:01.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:00 smithi181 ceph-mon[33998]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:01.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:16:01.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:00 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:01.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:01 smithi179 ceph-mon[29608]: Deploying daemon osd.1 on smithi179 2024-03-24T11:16:01.908 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:01 smithi179 ceph-mon[37907]: Deploying daemon osd.1 on smithi179 2024-03-24T11:16:02.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:01 smithi181 ceph-mon[33998]: Deploying daemon osd.1 on smithi179 2024-03-24T11:16:03.067 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:02 smithi179 ceph-mon[29608]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:03.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:02 smithi179 ceph-mon[37907]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:03.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:02 smithi181 ceph-mon[33998]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:04.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:04 smithi179 ceph-mon[29608]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:04.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:04 smithi179 ceph-mon[37907]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:05.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:04 smithi181 ceph-mon[33998]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:06 smithi181 ceph-mon[33998]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:06.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:06.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[29608]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:06 smithi179 ceph-mon[37907]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[29608]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[29608]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:16:09.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[37907]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:09.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:09.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:09.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.113 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:08 smithi179 ceph-mon[37907]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:16:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:08 smithi181 ceph-mon[33998]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:08 smithi181 ceph-mon[33998]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:16:09.518 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 1 on host 'smithi179' 2024-03-24T11:16:10.183 DEBUG:teuthology.orchestra.run.smithi179:osd.1> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:16:10.185 INFO:tasks.cephadm:Deploying osd.2 on smithi179 with /dev/vg_nvme/lv_2... 2024-03-24T11:16:10.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:10 smithi181 ceph-mon[33998]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:10.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:10.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:16:10.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:16:10.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[29608]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:10.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:10 smithi179 ceph-mon[37907]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:11.334 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1[47887]: 2024-03-24T11:16:11.053+0000 7fed60718700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T11:16:11.334 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1[47887]: 2024-03-24T11:16:11.056+0000 7fed58503700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:16:11.407 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:16:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:11 smithi181 ceph-mon[33998]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:16:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:11 smithi181 ceph-mon[33998]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:16:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:11 smithi181 ceph-mon[33998]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' 2024-03-24T11:16:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[37907]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:16:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[37907]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:16:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[37907]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' 2024-03-24T11:16:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[29608]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:16:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[29608]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:16:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:11 smithi179 ceph-mon[29608]: from='osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275]' entity='osd.1' 2024-03-24T11:16:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:12 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:16:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:12 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:16:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:12 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:12 smithi181 ceph-mon[33998]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:12 smithi181 ceph-mon[33998]: osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275] boot 2024-03-24T11:16:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:12 smithi181 ceph-mon[33998]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:16:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:12 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[37907]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[37907]: osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275] boot 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[37907]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:12.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:16:12.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:16:12.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:12.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[29608]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:16:12.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[29608]: osd.1 [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275] boot 2024-03-24T11:16:12.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[29608]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:16:12.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:12 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:16:14.016 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:16:14.016 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:16:14.016 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-24T11:16:14.016 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-24T11:16:14.016 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134492 s, 780 MB/s 2024-03-24T11:16:14.016 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:16:14.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:14.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:14.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:14.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:14.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:15.056 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_2 2024-03-24T11:16:15.293 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:16:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:15 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi179 2024-03-24T11:16:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:15 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi179 to 8032M 2024-03-24T11:16:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:15 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi179 2024-03-24T11:16:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:15 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi179 to 8032M 2024-03-24T11:16:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:15 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi179 2024-03-24T11:16:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:15 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi179 to 8032M 2024-03-24T11:16:16.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:16 smithi179 ceph-mon[29608]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:16 smithi179 ceph-mon[37907]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:17.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:16 smithi181 ceph-mon[33998]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:16:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:16:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:16:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:16:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:18.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:16:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:16:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:18.902 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:18 smithi179 ceph-mon[29608]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:18.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:18 smithi179 ceph-mon[29608]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:18.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:18 smithi179 ceph-mon[37907]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:18.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:18 smithi179 ceph-mon[37907]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:18 smithi181 ceph-mon[33998]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:18 smithi181 ceph-mon[33998]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3738849110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc"}]: dispatch 2024-03-24T11:16:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3738849110' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc"}]': finished 2024-03-24T11:16:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:19 smithi181 ceph-mon[33998]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:16:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:19 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3738849110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc"}]: dispatch 2024-03-24T11:16:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3738849110' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc"}]': finished 2024-03-24T11:16:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[29608]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:16:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3738849110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc"}]: dispatch 2024-03-24T11:16:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3738849110' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc"}]': finished 2024-03-24T11:16:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[37907]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:16:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:19 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:21.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:20 smithi181 ceph-mon[33998]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:20 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1595892678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:16:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:20 smithi179 ceph-mon[29608]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:20 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1595892678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:16:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:20 smithi179 ceph-mon[37907]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:20 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1595892678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:16:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:22 smithi179 ceph-mon[29608]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:22 smithi179 ceph-mon[37907]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:23.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:22 smithi181 ceph-mon[33998]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:25.082 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:24 smithi179 ceph-mon[29608]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:25.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:24 smithi179 ceph-mon[37907]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:24 smithi181 ceph-mon[33998]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:27.095 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:26 smithi179 ceph-mon[29608]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:27.095 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:26 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:16:27.095 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:26 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:27.095 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:26 smithi179 ceph-mon[37907]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:27.095 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:26 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:16:27.095 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:26 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:26 smithi181 ceph-mon[33998]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:26 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:16:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:26 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:28.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:27 smithi181 ceph-mon[33998]: Deploying daemon osd.2 on smithi179 2024-03-24T11:16:28.198 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:27 smithi179 ceph-mon[29608]: Deploying daemon osd.2 on smithi179 2024-03-24T11:16:28.198 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:27 smithi179 ceph-mon[37907]: Deploying daemon osd.2 on smithi179 2024-03-24T11:16:29.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:28 smithi181 ceph-mon[33998]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:28 smithi179 ceph-mon[29608]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:29.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:28 smithi179 ceph-mon[37907]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:30 smithi179 ceph-mon[29608]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:30.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:30 smithi179 ceph-mon[37907]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:30 smithi181 ceph-mon[33998]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:33.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:32 smithi181 ceph-mon[33998]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:32 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:32 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:32 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[29608]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:33.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[37907]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:32 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:34 smithi179 ceph-mon[29608]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:34.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:34 smithi179 ceph-mon[37907]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:34 smithi181 ceph-mon[33998]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:35.955 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 2 on host 'smithi179' 2024-03-24T11:16:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:16:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:16:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:16:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:36.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:36.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:36.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:36 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:36.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:36 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:36.579 DEBUG:teuthology.orchestra.run.smithi179:osd.2> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:16:36.582 INFO:tasks.cephadm:Deploying osd.3 on smithi179 with /dev/vg_nvme/lv_1... 2024-03-24T11:16:36.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:16:37.139 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[29608]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:16:37.139 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[29608]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:16:37.139 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:37.139 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[29608]: from='osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:37.139 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[29608]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:37.140 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[37907]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:16:37.140 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[37907]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:16:37.140 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:37.140 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[37907]: from='osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:37.140 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-mon[37907]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:37.140 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2[53906]: 2024-03-24T11:16:37.037+0000 7fd5f6d41700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T11:16:37.140 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2[53906]: 2024-03-24T11:16:37.040+0000 7fd5f2b34700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:16:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:16:37.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:37 smithi181 ceph-mon[33998]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:16:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:37 smithi181 ceph-mon[33998]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:16:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:37 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:37 smithi181 ceph-mon[33998]: from='osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:37 smithi181 ceph-mon[33998]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:16:38.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:38 smithi181 ceph-mon[33998]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:16:38.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:38 smithi181 ceph-mon[33998]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:16:38.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:38 smithi181 ceph-mon[33998]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:38.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[37907]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:16:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[37907]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:16:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[37907]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[29608]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:16:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[29608]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:16:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[29608]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:16:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:39.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:16:39.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:16:39.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[29608]: osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842] boot 2024-03-24T11:16:39.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[29608]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:16:39.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:39.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:16:39.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:16:39.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[37907]: osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842] boot 2024-03-24T11:16:39.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[37907]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:16:39.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:39 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:39.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:39 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:16:39.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:39 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:16:39.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:39 smithi181 ceph-mon[33998]: osd.2 [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842] boot 2024-03-24T11:16:39.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:39 smithi181 ceph-mon[33998]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:16:39.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:39 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:16:40.265 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:16:40.265 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:16:40.265 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-24T11:16:40.265 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-24T11:16:40.266 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130874 s, 801 MB/s 2024-03-24T11:16:40.266 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-24T11:16:40.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:40 smithi179 ceph-mon[37907]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:16:40.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:40 smithi179 ceph-mon[37907]: pgmap v80: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:40.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:16:40.317 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:40 smithi179 ceph-mon[29608]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:16:40.317 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:40 smithi179 ceph-mon[29608]: pgmap v80: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:40.317 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:16:40.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:40 smithi181 ceph-mon[33998]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:16:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:40 smithi181 ceph-mon[33998]: pgmap v80: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:16:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:16:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:16:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:16:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi179 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi179 to 5354M 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:16:41.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi179 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi179 to 5354M 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:41.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_1 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi179 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:16:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi179 to 5354M 2024-03-24T11:16:41.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:41.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:41.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:41.507 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:16:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:42 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:16:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:42 smithi179 ceph-mon[29608]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:16:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:42 smithi179 ceph-mon[29608]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:42.209 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57915]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:16:42.210 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57915]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:42.210 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57915]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:42.210 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57915]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:42 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:16:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:42 smithi179 ceph-mon[37907]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:16:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:42 smithi179 ceph-mon[37907]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:42.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:42 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:16:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:42 smithi181 ceph-mon[33998]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:16:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:42 smithi181 ceph-mon[33998]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:42.545 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57919]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:16:42.545 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57919]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:42.545 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57919]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:42.546 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57919]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:42.809 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57922]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:16:42.809 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57922]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:42.809 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57922]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:42.809 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57922]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:43.062 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57946]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:16:43.062 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57946]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:43.062 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57946]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:43.062 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:16:42 smithi179 sudo[57946]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:43.334 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 ceph-mon[29608]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:16:43.334 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:43.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:43 smithi179 ceph-mon[37907]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:16:43.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:43 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:43.335 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[57957]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:16:43.335 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[57957]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:43.335 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[57957]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:43.335 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[57957]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:43.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:43 smithi181 ceph-mon[33998]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:16:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:43 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:43.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58048]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:16:43.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58048]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:43.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58048]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:43.595 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58013]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:16:43.595 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:43.595 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58013]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:43.595 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58013]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:43.845 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58048]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:43.845 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58107]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:16:43.845 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58107]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:43.845 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58107]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:43.845 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:43 smithi179 sudo[58107]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:44.344 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36435]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:16:44.344 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36435]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:44.344 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36435]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:44.344 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36435]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:44.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:44.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:16:44.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:16:44.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:44.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:16:44.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:16:44.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:44.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36438]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36438]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36438]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 sudo[36438]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:44.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:44.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:44.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:44.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:44.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:44.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:44 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:44.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58131]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:16:44.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58131]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:44.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58131]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:44.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58131]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:45.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58155]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:16:45.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58155]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:16:45.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58155]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:16:45.158 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:44 smithi179 sudo[58155]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:16:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[29608]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:45 smithi179 ceph-mon[37907]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:16:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:45 smithi181 ceph-mon[33998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:16:46.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:46 smithi181 ceph-mon[33998]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1380982796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52"}]: dispatch 2024-03-24T11:16:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1380982796' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52"}]': finished 2024-03-24T11:16:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:46 smithi181 ceph-mon[33998]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:16:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:46 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[29608]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1380982796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52"}]: dispatch 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1380982796' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52"}]': finished 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[29608]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[37907]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1380982796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52"}]: dispatch 2024-03-24T11:16:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1380982796' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52"}]': finished 2024-03-24T11:16:46.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[37907]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T11:16:46.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:46 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:16:47.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1174558441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:16:47.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1174558441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:16:47.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1174558441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:16:48.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:48 smithi181 ceph-mon[33998]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-24T11:16:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:48 smithi179 ceph-mon[29608]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-24T11:16:48.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:48 smithi179 ceph-mon[37907]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-24T11:16:50.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:50 smithi179 ceph-mon[29608]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:16:50.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:50 smithi179 ceph-mon[37907]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:16:51.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:50 smithi181 ceph-mon[33998]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:16:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:52 smithi181 ceph-mon[33998]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:16:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:16:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:53.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:52 smithi179 ceph-mon[29608]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:16:53.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:16:53.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:52 smithi179 ceph-mon[37907]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:16:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:16:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:53.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:53 smithi179 ceph-mon[29608]: Deploying daemon osd.3 on smithi179 2024-03-24T11:16:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:53 smithi179 ceph-mon[37907]: Deploying daemon osd.3 on smithi179 2024-03-24T11:16:54.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:53 smithi181 ceph-mon[33998]: Deploying daemon osd.3 on smithi179 2024-03-24T11:16:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:54 smithi179 ceph-mon[29608]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:16:54.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:54 smithi179 ceph-mon[37907]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:16:55.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:54 smithi181 ceph-mon[33998]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:16:57.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:56 smithi181 ceph-mon[33998]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:16:57.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:56 smithi179 ceph-mon[29608]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:16:57.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:56 smithi179 ceph-mon[37907]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-24T11:16:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:58 smithi181 ceph-mon[33998]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-24T11:16:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:58 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:58 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:16:58 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[29608]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[37907]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:59.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:16:58 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:00.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:00 smithi179 ceph-mon[37907]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:00 smithi179 ceph-mon[29608]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:01.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:00 smithi181 ceph-mon[33998]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:01.894 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 3 on host 'smithi179' 2024-03-24T11:17:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:17:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:02.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:02.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:02.588 DEBUG:teuthology.orchestra.run.smithi179:osd.3> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T11:17:02.591 INFO:tasks.cephadm:Deploying osd.4 on smithi181 with /dev/vg_nvme/lv_4... 2024-03-24T11:17:02.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:17:02.836 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:17:03.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[37907]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:17:03.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[37907]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:17:03.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[37907]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:17:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:03.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[29608]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:17:03.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[29608]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:17:03.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[29608]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:17:03.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:03.210 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3[59886]: 2024-03-24T11:17:02.981+0000 7fdb5f182700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T11:17:03.211 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:17:02 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3[59886]: 2024-03-24T11:17:02.984+0000 7fdb5af75700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:17:03.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:02 smithi181 ceph-mon[33998]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:17:03.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:02 smithi181 ceph-mon[33998]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:17:03.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:02 smithi181 ceph-mon[33998]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-24T11:17:03.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:02 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:04.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[29608]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:17:04.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[29608]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:17:04.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:04.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:04.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[29608]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[37907]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:17:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[37907]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:17:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:03 smithi179 ceph-mon[37907]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:04.279 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:03 smithi181 ceph-mon[33998]: from='osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-24T11:17:04.279 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:03 smithi181 ceph-mon[33998]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:17:04.279 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:03 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:04.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:03 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:04.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:03 smithi181 ceph-mon[33998]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:17:04.652 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:17:04.653 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:17:04.653 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-24T11:17:04.653 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-24T11:17:04.653 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114391 s, 917 MB/s 2024-03-24T11:17:04.653 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-24T11:17:05.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:04 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:17:05.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:04 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:17:05.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:04 smithi181 ceph-mon[33998]: osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471] boot 2024-03-24T11:17:05.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:04 smithi181 ceph-mon[33998]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:17:05.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:04 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:05.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:04 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:17:05.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:04 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:17:05.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-03-24T11:17:05.350 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:17:05.350 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:17:05.350 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[29608]: osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471] boot 2024-03-24T11:17:05.350 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[29608]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[37907]: osd.3 [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471] boot 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[37907]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:17:05.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:04 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:17:05.498 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi179 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi179 to 4016M 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:06.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:06.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:05 smithi181 ceph-mon[33998]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:06.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:17:06.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi179 2024-03-24T11:17:06.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi179 to 4016M 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[29608]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi179 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi179 to 4016M 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:06.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:05 smithi179 ceph-mon[37907]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:07.952 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:07 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:07.953 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:07 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:07.953 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:07 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:07 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:07 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:07 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:07 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:07 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:07 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:08 smithi179 ceph-mon[29608]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:08 smithi179 ceph-mon[29608]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:08.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:08 smithi179 ceph-mon[37907]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:08 smithi179 ceph-mon[37907]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:08.970 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:08 smithi181 ceph-mon[33998]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:08.970 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:08 smithi181 ceph-mon[33998]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3537881774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]: dispatch 2024-03-24T11:17:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]: dispatch 2024-03-24T11:17:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]': finished 2024-03-24T11:17:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:09 smithi181 ceph-mon[33998]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:17:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:09 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3537881774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]: dispatch 2024-03-24T11:17:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]: dispatch 2024-03-24T11:17:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]': finished 2024-03-24T11:17:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[29608]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:17:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:10.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3537881774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]: dispatch 2024-03-24T11:17:10.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]: dispatch 2024-03-24T11:17:10.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176"}]': finished 2024-03-24T11:17:10.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[37907]: osdmap e28: 5 total, 4 up, 5 in 2024-03-24T11:17:10.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:09 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:11.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:10 smithi181 ceph-mon[33998]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:11.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:10 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/2693740266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:17:11.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:10 smithi179 ceph-mon[29608]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:11.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:10 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/2693740266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:17:11.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:10 smithi179 ceph-mon[37907]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:11.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:10 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/2693740266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:17:13.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:12 smithi181 ceph-mon[33998]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:13.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:12 smithi179 ceph-mon[29608]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:13.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:12 smithi179 ceph-mon[37907]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:15.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:14 smithi181 ceph-mon[33998]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:15.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:14 smithi179 ceph-mon[29608]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:14 smithi179 ceph-mon[37907]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:16 smithi181 ceph-mon[33998]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:17:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:16 smithi179 ceph-mon[29608]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:17:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:17.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:16 smithi179 ceph-mon[37907]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:17.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:17:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:17 smithi181 ceph-mon[33998]: Deploying daemon osd.4 on smithi181 2024-03-24T11:17:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:17 smithi179 ceph-mon[29608]: Deploying daemon osd.4 on smithi181 2024-03-24T11:17:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:17 smithi179 ceph-mon[37907]: Deploying daemon osd.4 on smithi181 2024-03-24T11:17:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:18 smithi181 ceph-mon[33998]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:19.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:18 smithi179 ceph-mon[29608]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:19.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:18 smithi179 ceph-mon[37907]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:20 smithi181 ceph-mon[33998]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:20 smithi179 ceph-mon[29608]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:20 smithi179 ceph-mon[37907]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[29608]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:22 smithi179 ceph-mon[37907]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:22.611 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:22 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:22.611 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:22 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:22.611 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:22 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:22.611 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:22 smithi181 ceph-mon[33998]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:24.431 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 4 on host 'smithi181' 2024-03-24T11:17:25.017 DEBUG:teuthology.orchestra.run.smithi181:osd.4> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T11:17:25.019 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-03-24T11:17:25.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:17:25.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:25.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:25.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:25.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.043 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:17:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:17:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:25.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:25.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:25.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:25.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:26.029 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:17:26.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-mon[33998]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:17:26.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-mon[33998]: from='osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:26.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-mon[33998]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:17:26.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:26.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-mon[33998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:26.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-mon[33998]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:26.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[29608]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:17:26.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[29608]: from='osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:26.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[29608]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:17:26.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:26.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[29608]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:26.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[29608]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[37907]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:17:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[37907]: from='osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[37907]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T11:17:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[37907]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:26 smithi179 ceph-mon[37907]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T11:17:26.611 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:17:26.515+0000 7fabb5506700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T11:17:26.611 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:17:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:17:26.520+0000 7fabad2f1700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: from='osd.4 ' entity='osd.4' 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415] boot 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:17:27.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:27 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: from='osd.4 ' entity='osd.4' 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415] boot 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T11:17:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: from='osd.4 ' entity='osd.4' 2024-03-24T11:17:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: osd.4 [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415] boot 2024-03-24T11:17:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: osdmap e31: 5 total, 5 up, 5 in 2024-03-24T11:17:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:27 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:17:28.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:28 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:17:28.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:28 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:17:28.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:28 smithi181 ceph-mon[33998]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:28.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:28 smithi181 ceph-mon[33998]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:17:28.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:17:28.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:17:28.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[29608]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:28.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[29608]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:17:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:17:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:17:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[37907]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:28 smithi179 ceph-mon[37907]: osdmap e32: 5 total, 5 up, 5 in 2024-03-24T11:17:28.506 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:17:28.506 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:17:28.506 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-24T11:17:28.507 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-24T11:17:28.507 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125373 s, 836 MB/s 2024-03-24T11:17:28.507 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-24T11:17:29.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-03-24T11:17:29.336 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:17:29.774 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi181 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi181 to 17088M 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.775 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:29 smithi181 ceph-mon[33998]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:17:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi181 2024-03-24T11:17:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi181 to 17088M 2024-03-24T11:17:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[37907]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi181 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi181 to 17088M 2024-03-24T11:17:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:29.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:29.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:29.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:29 smithi179 ceph-mon[29608]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T11:17:30.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:30 smithi181 ceph-mon[33998]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:30.831 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:30 smithi181 ceph-mon[33998]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:17:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:30 smithi179 ceph-mon[29608]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:30 smithi179 ceph-mon[29608]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:17:30.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:30 smithi179 ceph-mon[37907]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:30.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:30 smithi179 ceph-mon[37907]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T11:17:31.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:31 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:31.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:31 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:31.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:31 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:31.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:31 smithi181 ceph-mon[33998]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:17:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[29608]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:17:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:31 smithi179 ceph-mon[37907]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:17:32.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:32 smithi181 ceph-mon[33998]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:32.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:32 smithi181 ceph-mon[33998]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:32 smithi179 ceph-mon[29608]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:32 smithi179 ceph-mon[29608]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:32.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:32 smithi179 ceph-mon[37907]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:32.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:32 smithi179 ceph-mon[37907]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 523 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3830978484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]: dispatch 2024-03-24T11:17:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:33 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]: dispatch 2024-03-24T11:17:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:33 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]': finished 2024-03-24T11:17:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:33 smithi181 ceph-mon[33998]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:17:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:33 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3830978484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]: dispatch 2024-03-24T11:17:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]: dispatch 2024-03-24T11:17:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]': finished 2024-03-24T11:17:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[29608]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:17:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3830978484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]: dispatch 2024-03-24T11:17:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]: dispatch 2024-03-24T11:17:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646"}]': finished 2024-03-24T11:17:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[37907]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T11:17:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:33 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:35.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:34 smithi181 ceph-mon[33998]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-24T11:17:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:34 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/510125881' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:17:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:34 smithi179 ceph-mon[29608]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-24T11:17:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:34 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/510125881' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:17:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:34 smithi179 ceph-mon[37907]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-24T11:17:35.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:34 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/510125881' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:17:36.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:36 smithi181 ceph-mon[33998]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:17:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:36 smithi179 ceph-mon[29608]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:17:37.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:36 smithi179 ceph-mon[37907]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:17:37.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:37 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:17:37.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:37 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:17:37.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:37 smithi181 ceph-mon[33998]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:17:37.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:37 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:17:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:17:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[29608]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:17:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:17:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:17:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[37907]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:17:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:37 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:38.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:38 smithi181 ceph-mon[33998]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-24T11:17:39.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:38 smithi179 ceph-mon[29608]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-24T11:17:39.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:38 smithi179 ceph-mon[37907]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-24T11:17:41.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:40 smithi181 ceph-mon[33998]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:17:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:17:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:40 smithi179 ceph-mon[29608]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:17:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:17:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:40 smithi179 ceph-mon[37907]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:17:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:17:41.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:41 smithi181 ceph-mon[33998]: Deploying daemon osd.5 on smithi181 2024-03-24T11:17:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:41 smithi179 ceph-mon[29608]: Deploying daemon osd.5 on smithi181 2024-03-24T11:17:42.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:41 smithi179 ceph-mon[37907]: Deploying daemon osd.5 on smithi181 2024-03-24T11:17:43.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:42 smithi181 ceph-mon[33998]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:42 smithi179 ceph-mon[29608]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:43.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:42 smithi179 ceph-mon[37907]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:45.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:44 smithi181 ceph-mon[33998]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:45.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:44 smithi179 ceph-mon[29608]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:45.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:44 smithi179 ceph-mon[37907]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:46.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:46 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:46.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:46 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:46.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:46 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:46.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:46 smithi181 ceph-mon[33998]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:46.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:46 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:17:46.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:46 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:17:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[29608]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:17:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:17:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[37907]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:17:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:46 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:17:47.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:47 smithi181 ceph-mon[33998]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:17:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:47 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:47 smithi179 ceph-mon[29608]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:17:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:47 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:47.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:47 smithi179 ceph-mon[37907]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:17:47.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:47 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:48.775 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:49.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.080 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:48 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:17:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:17:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:48 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:49.403 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T11:17:49.405 INFO:tasks.cephadm:Deploying osd.6 on smithi181 with /dev/vg_nvme/lv_2... 2024-03-24T11:17:49.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:17:50.072 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:17:50.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-mon[33998]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:17:50.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-mon[33998]: from='osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:50.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-mon[33998]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:17:50.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:50.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-mon[33998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:50.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-mon[33998]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:50.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[29608]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:17:50.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[29608]: from='osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:50.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[29608]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:17:50.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[29608]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[29608]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[37907]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[37907]: from='osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[37907]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[37907]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:17:50.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:50 smithi179 ceph-mon[37907]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T11:17:50.805 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:17:50.587+0000 7fb2d557a700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T11:17:50.805 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:17:50 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:17:50.595+0000 7fb2cd365700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:17:51.390 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:51 smithi181 ceph-mon[33998]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:17:51.390 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:51 smithi181 ceph-mon[33998]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T11:17:51.390 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:51 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:51.390 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:51 smithi181 ceph-mon[33998]: from='osd.5 ' entity='osd.5' 2024-03-24T11:17:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[29608]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:17:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[29608]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T11:17:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[29608]: from='osd.5 ' entity='osd.5' 2024-03-24T11:17:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[37907]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:17:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[37907]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T11:17:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:51 smithi179 ceph-mon[37907]: from='osd.5 ' entity='osd.5' 2024-03-24T11:17:52.241 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:52 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:17:52.241 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:52 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:17:52.241 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:52 smithi181 ceph-mon[33998]: osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782] boot 2024-03-24T11:17:52.241 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:52 smithi181 ceph-mon[33998]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:17:52.241 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:52.241 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:52 smithi181 ceph-mon[33998]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:52.241 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:52 smithi181 ceph-mon[33998]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:17:52.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:17:52.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:17:52.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[29608]: osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782] boot 2024-03-24T11:17:52.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[29608]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:17:52.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:52.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[29608]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[29608]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[37907]: osd.5 [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782] boot 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[37907]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[37907]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:52 smithi179 ceph-mon[37907]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:17:52.790 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:17:52.790 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:17:52.790 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-24T11:17:52.790 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-24T11:17:52.790 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124032 s, 845 MB/s 2024-03-24T11:17:52.790 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-24T11:17:53.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-03-24T11:17:53.953 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi181 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi181 to 8544M 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:54.325 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:17:54.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: pgmap v141: 1 pgs: 1 remapped; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:54.326 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:54 smithi181 ceph-mon[33998]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:17:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi181 2024-03-24T11:17:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi181 to 8544M 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: pgmap v141: 1 pgs: 1 remapped; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[37907]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi181 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi181 to 8544M 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: pgmap v141: 1 pgs: 1 remapped; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:54 smithi179 ceph-mon[29608]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:17:56.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:55 smithi181 ceph-mon[33998]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:17:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:55 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:17:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:55 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:17:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:55 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:55 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:55 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[37907]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:17:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:17:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:17:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:56.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[29608]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:17:56.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:17:56.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:17:56.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:17:56.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:17:56.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:55 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:57.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:56 smithi181 ceph-mon[33998]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:57.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:56 smithi181 ceph-mon[33998]: pgmap v144: 1 pgs: 1 remapped; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:57.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:56 smithi179 ceph-mon[29608]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:57.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:56 smithi179 ceph-mon[29608]: pgmap v144: 1 pgs: 1 remapped; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:57.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:56 smithi179 ceph-mon[37907]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:57.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:56 smithi179 ceph-mon[37907]: pgmap v144: 1 pgs: 1 remapped; 577 KiB data, 246 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:17:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:58 smithi181 ceph-mon[33998]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T11:17:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/973793965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]: dispatch 2024-03-24T11:17:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:58 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]: dispatch 2024-03-24T11:17:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:58 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]': finished 2024-03-24T11:17:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:58 smithi181 ceph-mon[33998]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:17:59.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:58 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:17:59.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[29608]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T11:17:59.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/973793965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]: dispatch 2024-03-24T11:17:59.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]: dispatch 2024-03-24T11:17:59.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]': finished 2024-03-24T11:17:59.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[29608]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:17:59.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:17:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[37907]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T11:17:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/973793965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]: dispatch 2024-03-24T11:17:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]: dispatch 2024-03-24T11:17:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793"}]': finished 2024-03-24T11:17:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[37907]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T11:17:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:58 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:00.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:17:59 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3785152106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:18:00.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:17:59 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3785152106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:18:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:17:59 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3785152106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:18:01.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:00 smithi181 ceph-mon[33998]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:01.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:00 smithi179 ceph-mon[29608]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:01.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:00 smithi179 ceph-mon[37907]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:02.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:01 smithi181 ceph-mon[33998]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:18:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:01 smithi179 ceph-mon[29608]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:18:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:01 smithi179 ceph-mon[37907]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:18:05.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:04 smithi181 ceph-mon[33998]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:18:05.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:04 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:18:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:04 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:05.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:04 smithi179 ceph-mon[29608]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:18:05.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:04 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:18:05.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:04 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:04 smithi179 ceph-mon[37907]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-24T11:18:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:04 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T11:18:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:04 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:05.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:05 smithi181 ceph-mon[33998]: Deploying daemon osd.6 on smithi181 2024-03-24T11:18:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:05 smithi179 ceph-mon[37907]: Deploying daemon osd.6 on smithi181 2024-03-24T11:18:06.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:05 smithi179 ceph-mon[29608]: Deploying daemon osd.6 on smithi181 2024-03-24T11:18:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:06 smithi181 ceph-mon[33998]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:18:07.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:06 smithi179 ceph-mon[29608]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:18:07.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:06 smithi179 ceph-mon[37907]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T11:18:09.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:08 smithi181 ceph-mon[33998]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:09.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:08 smithi179 ceph-mon[29608]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:09.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:08 smithi179 ceph-mon[37907]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:10 smithi181 ceph-mon[33998]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:11.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[29608]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:11.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:11.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:11.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:11.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[37907]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:11.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:11.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:11.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:12.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:11 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:18:12.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:11 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:18:12.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:11 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:18:12.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:11 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:18:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:11 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:18:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:11 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:18:13.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:12 smithi181 ceph-mon[33998]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:13.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:12 smithi181 ceph-mon[33998]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T11:18:13.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:12 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:13.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:12 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:13.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:12 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:13.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[29608]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:13.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[29608]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T11:18:13.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:13.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:13.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[37907]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[37907]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T11:18:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:12 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:13.429 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 6 on host 'smithi181' 2024-03-24T11:18:13.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:13 smithi181 ceph-mon[33998]: from='osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:18:13.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:13 smithi181 ceph-mon[33998]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:18:13.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:13.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:13 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:14.018 DEBUG:teuthology.orchestra.run.smithi181:osd.6> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T11:18:14.020 INFO:tasks.cephadm:Deploying osd.7 on smithi181 with /dev/vg_nvme/lv_1... 2024-03-24T11:18:14.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T11:18:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[29608]: from='osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:18:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[29608]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:18:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[37907]: from='osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:18:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[37907]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T11:18:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:13 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:15.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:14 smithi181 ceph-mon[33998]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:15.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:14 smithi181 ceph-mon[33998]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:18:15.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:14 smithi181 ceph-mon[33998]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T11:18:15.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:14 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:15.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:14 smithi181 ceph-mon[33998]: from='osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:15.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:14 smithi181 ceph-mon[33998]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:15.092 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:18:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[37907]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[37907]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:18:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[37907]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T11:18:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[37907]: from='osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[37907]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:15.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[29608]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:15.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[29608]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T11:18:15.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[29608]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T11:18:15.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:15.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[29608]: from='osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:15.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:14 smithi179 ceph-mon[29608]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:15.646 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:18:15.265+0000 7f0a881d4700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T11:18:15.647 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:18:15.272+0000 7f0a7ffbf700 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:18:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:18:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T11:18:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: from='osd.6 ' entity='osd.6' 2024-03-24T11:18:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374] boot 2024-03-24T11:18:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:18:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:15 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: from='osd.6 ' entity='osd.6' 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374] boot 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:18:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: from='osd.6 ' entity='osd.6' 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: osd.6 [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374] boot 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: osdmap e50: 7 total, 7 up, 7 in 2024-03-24T11:18:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:15 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:18:16.932 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:16 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:18:16.933 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:16 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:18:16.933 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:16 smithi181 ceph-mon[33998]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:16.933 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:16 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:16.933 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:16 smithi181 ceph-mon[33998]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:18:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:18:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:18:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[29608]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[29608]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:18:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:18:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:18:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[37907]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T11:18:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:16 smithi179 ceph-mon[37907]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T11:18:18.030 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T11:18:18.030 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T11:18:18.030 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-03-24T11:18:18.030 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-03-24T11:18:18.030 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124878 s, 840 MB/s 2024-03-24T11:18:18.030 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-03-24T11:18:18.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi181 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi181 to 5696M 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:18.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi181 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi181 to 5696M 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi181 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:18.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi181 to 5696M 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:18.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:18.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-03-24T11:18:18.864 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:18:18.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:18 smithi181 ceph-mon[33998]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-03-24T11:18:18.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:18 smithi181 ceph-mon[33998]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T11:18:19.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:18 smithi179 ceph-mon[37907]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-03-24T11:18:19.258 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:18 smithi179 ceph-mon[37907]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T11:18:19.259 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:18 smithi179 ceph-mon[29608]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-03-24T11:18:19.259 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:18 smithi179 ceph-mon[29608]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T11:18:20.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:19 smithi181 ceph-mon[33998]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T11:18:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:19 smithi179 ceph-mon[29608]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T11:18:20.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:19 smithi179 ceph-mon[37907]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T11:18:21.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:20 smithi181 ceph-mon[33998]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-03-24T11:18:21.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:20 smithi181 ceph-mon[33998]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T11:18:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:18:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:18:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:20 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[37907]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[37907]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[29608]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[29608]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:18:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:18:21.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:20 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:21 smithi181 ceph-mon[33998]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:22.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:21 smithi179 ceph-mon[29608]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:21 smithi179 ceph-mon[37907]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:23.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:22 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:18:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:22 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:18:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:22 smithi181 ceph-mon[33998]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-24T11:18:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:22 smithi181 ceph-mon[33998]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[29608]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[29608]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[37907]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-24T11:18:23.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:22 smithi179 ceph-mon[37907]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T11:18:24.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/2250020665' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]: dispatch 2024-03-24T11:18:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]: dispatch 2024-03-24T11:18:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]': finished 2024-03-24T11:18:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:23 smithi181 ceph-mon[33998]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T11:18:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:23 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3857920494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:18:24.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/2250020665' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]: dispatch 2024-03-24T11:18:24.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]: dispatch 2024-03-24T11:18:24.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]': finished 2024-03-24T11:18:24.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[29608]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3857920494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/2250020665' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]: dispatch 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]: dispatch 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00"}]': finished 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[37907]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:24.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3857920494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:18:25.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:24 smithi181 ceph-mon[33998]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-24T11:18:25.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:24 smithi179 ceph-mon[29608]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-24T11:18:25.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:24 smithi179 ceph-mon[37907]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-24T11:18:27.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:26 smithi181 ceph-mon[33998]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:26 smithi179 ceph-mon[29608]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:26 smithi179 ceph-mon[37907]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:29.070 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:28 smithi181 ceph-mon[33998]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:18:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:28 smithi179 ceph-mon[29608]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:18:29.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:28 smithi179 ceph-mon[37907]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-24T11:18:30.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:18:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:29 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:18:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:29 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T11:18:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:29 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:31.114 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:30 smithi181 ceph-mon[33998]: Deploying daemon osd.7 on smithi181 2024-03-24T11:18:31.114 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:30 smithi181 ceph-mon[33998]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:30 smithi179 ceph-mon[29608]: Deploying daemon osd.7 on smithi181 2024-03-24T11:18:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:30 smithi179 ceph-mon[29608]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:31.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:30 smithi179 ceph-mon[37907]: Deploying daemon osd.7 on smithi181 2024-03-24T11:18:31.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:30 smithi179 ceph-mon[37907]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:32.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:31 smithi179 ceph-mon[29608]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:31 smithi179 ceph-mon[37907]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:32.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:31 smithi181 ceph-mon[33998]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:35.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:34 smithi181 ceph-mon[33998]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:35.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:34 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:35.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:34 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:35.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:34 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[29608]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[37907]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:35.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:35.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:35.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:34 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:37.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:36 smithi181 ceph-mon[33998]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:36 smithi179 ceph-mon[29608]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:36 smithi179 ceph-mon[37907]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:37.693 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 7 on host 'smithi181' 2024-03-24T11:18:38.249 DEBUG:teuthology.orchestra.run.smithi181:osd.7> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T11:18:38.251 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-24T11:18:38.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:18:38.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:38.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:38.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:18:38.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:18:38.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:38.277 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:38 smithi181 ceph-mon[33998]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:18:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[37907]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:38.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:38.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:38 smithi179 ceph-mon[29608]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:38.499 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:39.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[29608]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:18:39.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[29608]: from='osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:39.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[29608]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T11:18:39.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:39.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[29608]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:39.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[37907]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:18:39.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[37907]: from='osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:39.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[37907]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T11:18:39.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:39.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:39 smithi179 ceph-mon[37907]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:39.332 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:39 smithi181 ceph-mon[33998]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T11:18:39.332 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:39 smithi181 ceph-mon[33998]: from='osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:39.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:39 smithi181 ceph-mon[33998]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T11:18:39.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:39 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:39.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:39 smithi181 ceph-mon[33998]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-03-24T11:18:39.608 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:18:39 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:18:39.517+0000 7f14c9405700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T11:18:39.608 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:18:39 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:18:39.525+0000 7f14c11f0700 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: from='osd.7 ' entity='osd.7' 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117] boot 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:18:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: from='osd.7 ' entity='osd.7' 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117] boot 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:18:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:40 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.375 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:18:40.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: from='osd.7 ' entity='osd.7' 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: osd.7 [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117] boot 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: osdmap e59: 8 total, 8 up, 8 in 2024-03-24T11:18:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:40 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:18:40.991 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1711279120,"num_in_osds":8,"osd_in_since":1711279103,"num_remapped_pgs":1} 2024-03-24T11:18:40.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:18:41.242 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:41.365 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:18:41.365 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:18:41.365 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/236601336' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:18:41.365 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.365 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.365 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:18:41.366 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/236601336' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:41.367 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:41 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/236601336' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:41.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:41 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:42 smithi181 ceph-mon[33998]: Detected new or changed devices on smithi181 2024-03-24T11:18:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:42 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi181 to 4272M 2024-03-24T11:18:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:42 smithi181 ceph-mon[33998]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:18:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:42 smithi181 ceph-mon[33998]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:42.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[37907]: Detected new or changed devices on smithi181 2024-03-24T11:18:42.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi181 to 4272M 2024-03-24T11:18:42.430 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[37907]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:18:42.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[37907]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:42.431 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[29608]: Detected new or changed devices on smithi181 2024-03-24T11:18:42.431 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi181 to 4272M 2024-03-24T11:18:42.431 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[29608]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T11:18:42.431 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:42 smithi179 ceph-mon[29608]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:43.188 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:18:43.189 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":62,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:13:18.639983+0000","modified":"2024-03-24T11:18:43.107052+0000","last_up_change":"2024-03-24T11:18:40.079955+0000","last_in_change":"2024-03-24T11:18:23.090476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:16:40.018600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14f300f0-7498-4ead-a87e-d9f38bdff21b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6803","nonce":1725503377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6805","nonce":1725503377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6809","nonce":1725503377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6807","nonce":1725503377}]},"public_addr":"172.21.15.179:6803/1725503377","cluster_addr":"172.21.15.179:6805/1725503377","heartbeat_back_addr":"172.21.15.179:6809/1725503377","heartbeat_front_addr":"172.21.15.179:6807/1725503377","state":["exists","up"]},{"osd":1,"uuid":"3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6811","nonce":2564847275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6813","nonce":2564847275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6817","nonce":2564847275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6815","nonce":2564847275}]},"public_addr":"172.21.15.179:6811/2564847275","cluster_addr":"172.21.15.179:6813/2564847275","heartbeat_back_addr":"172.21.15.179:6817/2564847275","heartbeat_front_addr":"172.21.15.179:6815/2564847275","state":["exists","up"]},{"osd":2,"uuid":"863b92b8-c68c-41a3-aed4-fd79e96d34dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6819","nonce":3297713842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6821","nonce":3297713842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6825","nonce":3297713842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6823","nonce":3297713842}]},"public_addr":"172.21.15.179:6819/3297713842","cluster_addr":"172.21.15.179:6821/3297713842","heartbeat_back_addr":"172.21.15.179:6825/3297713842","heartbeat_front_addr":"172.21.15.179:6823/3297713842","state":["exists","up"]},{"osd":3,"uuid":"c3d6b564-195d-4771-9aae-93a9e2eaba52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6827","nonce":944142471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6829","nonce":944142471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6833","nonce":944142471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6831","nonce":944142471}]},"public_addr":"172.21.15.179:6827/944142471","cluster_addr":"172.21.15.179:6829/944142471","heartbeat_back_addr":"172.21.15.179:6833/944142471","heartbeat_front_addr":"172.21.15.179:6831/944142471","state":["exists","up"]},{"osd":4,"uuid":"91bf5303-3a94-407d-aef1-e43bf0e16176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6801","nonce":3231660415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6803","nonce":3231660415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6807","nonce":3231660415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6805","nonce":3231660415}]},"public_addr":"172.21.15.181:6801/3231660415","cluster_addr":"172.21.15.181:6803/3231660415","heartbeat_back_addr":"172.21.15.181:6807/3231660415","heartbeat_front_addr":"172.21.15.181:6805/3231660415","state":["exists","up"]},{"osd":5,"uuid":"cba3f489-e8be-46f7-a281-e69a0ad08646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6809","nonce":775575782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6811","nonce":775575782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6815","nonce":775575782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6813","nonce":775575782}]},"public_addr":"172.21.15.181:6809/775575782","cluster_addr":"172.21.15.181:6811/775575782","heartbeat_back_addr":"172.21.15.181:6815/775575782","heartbeat_front_addr":"172.21.15.181:6813/775575782","state":["exists","up"]},{"osd":6,"uuid":"44a2c3a0-b704-4c10-a257-c54de47ee793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6817","nonce":2357509374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6819","nonce":2357509374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6823","nonce":2357509374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6821","nonce":2357509374}]},"public_addr":"172.21.15.181:6817/2357509374","cluster_addr":"172.21.15.181:6819/2357509374","heartbeat_back_addr":"172.21.15.181:6823/2357509374","heartbeat_front_addr":"172.21.15.181:6821/2357509374","state":["exists","up"]},{"osd":7,"uuid":"df60c4b4-d2f2-4e1f-8e1f-3169c403de00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6825","nonce":732039117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6827","nonce":732039117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6831","nonce":732039117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6829","nonce":732039117}]},"public_addr":"172.21.15.181:6825/732039117","cluster_addr":"172.21.15.181:6827/732039117","heartbeat_back_addr":"172.21.15.181:6831/732039117","heartbeat_front_addr":"172.21.15.181:6829/732039117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:21.394931+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:42.031740+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:17:45.946506+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:11.255163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:42.268864+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:17:49.171368+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:14.062509+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:38.510933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2066930919":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/3344305045":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2047439415":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2748363950":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/883221702":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2108875546":"2024-03-25T11:14:17.688945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:18:43.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:43 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:18:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:43 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:18:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:43 smithi181 ceph-mon[33998]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:18:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:43 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:18:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:43 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:18:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:18:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:18:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[37907]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:18:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:18:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:18:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:18:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:18:43.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[29608]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T11:18:43.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:18:43.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:43 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:18:43.951 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:16:40.018600+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T11:18:43.951 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:18:44.191 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:44.313 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:44 smithi179 ceph-mon[29608]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:18:44.313 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1979196091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:18:44.313 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:44 smithi179 ceph-mon[29608]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:44.314 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:44 smithi179 ceph-mon[37907]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:18:44.314 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1979196091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:18:44.314 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:44 smithi179 ceph-mon[37907]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:44.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:44 smithi181 ceph-mon[33998]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T11:18:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1979196091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:18:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:44 smithi181 ceph-mon[33998]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:45.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:45 smithi181 ceph-mon[33998]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:18:45.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:45 smithi179 ceph-mon[29608]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:18:45.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:45 smithi179 ceph-mon[37907]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T11:18:46.141 INFO:teuthology.orchestra.run.smithi179.stdout:pg_num: 1 2024-03-24T11:18:46.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:46 smithi181 ceph-mon[33998]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-24T11:18:46.447 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:46 smithi179 ceph-mon[29608]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-24T11:18:46.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:46 smithi179 ceph-mon[37907]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-24T11:18:46.748 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi179 2024-03-24T11:18:46.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi179=foo.a' 2024-03-24T11:18:46.998 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:18:47.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2138795994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:18:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2138795994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:18:47.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2138795994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:18:48.196 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:48 smithi181 ceph-mon[33998]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:48 smithi179 ceph-mon[29608]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:48.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:48 smithi179 ceph-mon[37907]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:18:48.650 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled rgw.foo.a update... 2024-03-24T11:18:49.178 DEBUG:teuthology.orchestra.run.smithi179:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-24T11:18:49.180 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi181 2024-03-24T11:18:49.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-24T11:18:49.428 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi179=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: Saving service rgw.foo.a spec with placement smithi179=foo.a;count:1 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:18:49.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:18:49.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:49 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:49.942 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi179=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:49.942 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: Saving service rgw.foo.a spec with placement smithi179=foo.a;count:1 2024-03-24T11:18:49.942 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:49.943 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi179=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: Saving service rgw.foo.a spec with placement smithi179=foo.a;count:1 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:49.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:49 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:50.669 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:50 smithi181 ceph-mon[33998]: Deploying daemon rgw.foo.a on smithi179 2024-03-24T11:18:50.669 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:50 smithi181 ceph-mon[33998]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-24T11:18:50.853 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:50 smithi179 ceph-mon[29608]: Deploying daemon rgw.foo.a on smithi179 2024-03-24T11:18:50.854 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:50 smithi179 ceph-mon[29608]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-24T11:18:50.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:50 smithi179 ceph-mon[37907]: Deploying daemon rgw.foo.a on smithi179 2024-03-24T11:18:50.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:50 smithi179 ceph-mon[37907]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-24T11:18:51.459 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:18:51 smithi179 systemd[1]: Starting Ceph rgw.foo.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:18:51.664 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'datapool' created 2024-03-24T11:18:51.825 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/1693996328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:18:51.825 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:51 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:18:51.826 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/1693996328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:18:51.826 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:51 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:18:51.826 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:18:51 smithi179 podman[64560]: 2024-03-24 11:18:51.547548603 +0000 UTC m=+0.107181540 container create bec3ddf95fe80ca71f44af4efbaa227d4d746b08116e31afa4c91e121bb07c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T11:18:51.826 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:18:51 smithi179 podman[64560]: 2024-03-24 11:18:51.458068165 +0000 UTC m=+0.017701114 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:18:51.826 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:18:51 smithi179 podman[64560]: 2024-03-24 11:18:51.741985046 +0000 UTC m=+0.301617984 container init bec3ddf95fe80ca71f44af4efbaa227d4d746b08116e31afa4c91e121bb07c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-db0330b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T11:18:51.826 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:18:51 smithi179 podman[64560]: 2024-03-24 11:18:51.744775988 +0000 UTC m=+0.304408935 container start bec3ddf95fe80ca71f44af4efbaa227d4d746b08116e31afa4c91e121bb07c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T11:18:51.826 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:18:51 smithi179 bash[64560]: bec3ddf95fe80ca71f44af4efbaa227d4d746b08116e31afa4c91e121bb07c0d 2024-03-24T11:18:51.954 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/1693996328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:18:51.954 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:51 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T11:18:52.209 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:18:51 smithi179 systemd[1]: Started Ceph rgw.foo.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:18:52.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- rbd pool init datapool 2024-03-24T11:18:52.507 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:18:52.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:18:52.899 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:18:52.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: pgmap v191: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:18:52.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.900 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:52 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:18:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:18:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: pgmap v191: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:18:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: pgmap v191: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:52 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:53.806 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: Saving service rgw.foo.a spec with placement smithi179=foo.a;count:1 2024-03-24T11:18:53.806 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:18:53.806 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:18:53.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4018859010' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:18:53.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:18:53.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:53.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:18:53.807 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:53 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: Saving service rgw.foo.a spec with placement smithi179=foo.a;count:1 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4018859010' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:18:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: Saving service rgw.foo.a spec with placement smithi179=foo.a;count:1 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4018859010' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:18:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:53 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:18:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:18:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: Checking dashboard <-> RGW credentials 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3345558381' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: Checking dashboard <-> RGW credentials 2024-03-24T11:18:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3345558381' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:54 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:18:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T11:18:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:55.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: Checking dashboard <-> RGW credentials 2024-03-24T11:18:55.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: pgmap v194: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:18:55.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3345558381' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:18:55.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T11:18:55.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T11:18:55.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T11:18:55.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:55.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:55.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:55.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:54 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:18:56.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:18:56.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:18:56.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[29608]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:18:56.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[29608]: pgmap v197: 68 pgs: 7 creating+peering, 15 active+clean, 46 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:18:56.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:18:56.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:18:56.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[37907]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:18:56.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:56 smithi179 ceph-mon[37907]: pgmap v197: 68 pgs: 7 creating+peering, 15 active+clean, 46 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:18:57.061 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:56 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:18:57.062 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:56 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:18:57.062 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:56 smithi181 ceph-mon[33998]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T11:18:57.062 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:56 smithi181 ceph-mon[33998]: pgmap v197: 68 pgs: 7 creating+peering, 15 active+clean, 46 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:18:57.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.181 --placement '1;smithi181=iscsi.a' 2024-03-24T11:18:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:57 smithi181 ceph-mon[33998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:57 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:57 smithi181 ceph-mon[33998]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:57 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:57 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[29608]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[29608]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[37907]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:18:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:18:57.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[37907]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T11:18:57.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:57.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:57 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:18:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:58 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:18:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:58 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:18:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:58 smithi181 ceph-mon[33998]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:18:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:58 smithi181 ceph-mon[33998]: pgmap v200: 100 pgs: 2 creating+activating, 18 creating+peering, 58 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T11:18:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:18:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:18:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[29608]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:18:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[29608]: pgmap v200: 100 pgs: 2 creating+activating, 18 creating+peering, 58 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T11:18:58.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:18:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:18:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[37907]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T11:18:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:58 smithi179 ceph-mon[37907]: pgmap v200: 100 pgs: 2 creating+activating, 18 creating+peering, 58 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-24T11:18:59.313 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled iscsi.datapool update... 2024-03-24T11:18:59.807 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-24T11:18:59.807 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:18:59.807 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T11:18:59.841 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:18:59.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T11:18:59.871 DEBUG:teuthology.orchestra.run.smithi181:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@iscsi.iscsi.a.service 2024-03-24T11:18:59.914 INFO:tasks.cephadm:Adding prometheus.a on smithi181 2024-03-24T11:18:59.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi181=a' 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[29608]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[37907]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:18:59 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:18:59.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:59 smithi181 ceph-mon[33998]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T11:18:59.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:59 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:59 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:59 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:59 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:18:59.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:18:59 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:00.187 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: pgmap v203: 132 pgs: 2 creating+activating, 20 creating+peering, 64 active+clean, 46 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T11:19:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:19:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:19:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:00 smithi181 ceph-mon[33998]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:19:01.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:01.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-03-24T11:19:01.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:19:01.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: pgmap v203: 132 pgs: 2 creating+activating, 20 creating+peering, 64 active+clean, 46 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[29608]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:19:01.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/729929273' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3343601211' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: pgmap v203: 132 pgs: 2 creating+activating, 20 creating+peering, 64 active+clean, 46 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:19:01.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:00 smithi179 ceph-mon[37907]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T11:19:02.033 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled prometheus update... 2024-03-24T11:19:02.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:19:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:19:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: Deploying daemon iscsi.iscsi.a on smithi181 2024-03-24T11:19:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:01 smithi181 ceph-mon[33998]: pgmap v205: 132 pgs: 7 creating+activating, 13 creating+peering, 99 active+clean, 13 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1.5 KiB/s wr, 31 op/s 2024-03-24T11:19:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: Deploying daemon iscsi.iscsi.a on smithi181 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[29608]: pgmap v205: 132 pgs: 7 creating+activating, 13 creating+peering, 99 active+clean, 13 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1.5 KiB/s wr, 31 op/s 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T11:19:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T11:19:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: Deploying daemon iscsi.iscsi.a on smithi181 2024-03-24T11:19:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:01 smithi179 ceph-mon[37907]: pgmap v205: 132 pgs: 7 creating+activating, 13 creating+peering, 99 active+clean, 13 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1.5 KiB/s wr, 31 op/s 2024-03-24T11:19:02.583 DEBUG:teuthology.orchestra.run.smithi181:prometheus.a> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T11:19:02.626 INFO:tasks.cephadm:Adding node-exporter.a on smithi179 2024-03-24T11:19:02.626 INFO:tasks.cephadm:Adding node-exporter.b on smithi181 2024-03-24T11:19:02.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi179=a;smithi181=b' 2024-03-24T11:19:03.353 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:03 smithi181 ceph-mon[33998]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:03.353 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:03 smithi181 ceph-mon[33998]: Saving service prometheus spec with placement smithi181=a;count:1 2024-03-24T11:19:03.353 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:03 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:03.419 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:19:03.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:03 smithi179 ceph-mon[29608]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:03.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:03 smithi179 ceph-mon[29608]: Saving service prometheus spec with placement smithi181=a;count:1 2024-03-24T11:19:03.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:03 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:03 smithi179 ceph-mon[37907]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:03 smithi179 ceph-mon[37907]: Saving service prometheus spec with placement smithi181=a;count:1 2024-03-24T11:19:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:03 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:04.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:03 smithi181 systemd[1]: Starting Ceph iscsi.iscsi.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:19:04.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:04 smithi181 ceph-mon[33998]: pgmap v206: 132 pgs: 7 creating+activating, 2 creating+peering, 123 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.8 KiB/s wr, 132 op/s 2024-03-24T11:19:04.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:04 smithi181 ceph-mon[33998]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:19:04.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:04 smithi179 ceph-mon[29608]: pgmap v206: 132 pgs: 7 creating+activating, 2 creating+peering, 123 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.8 KiB/s wr, 132 op/s 2024-03-24T11:19:04.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:04 smithi179 ceph-mon[29608]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:19:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:04 smithi179 ceph-mon[37907]: pgmap v206: 132 pgs: 7 creating+activating, 2 creating+peering, 123 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.8 KiB/s wr, 132 op/s 2024-03-24T11:19:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:04 smithi179 ceph-mon[37907]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T11:19:05.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:04 smithi181 podman[61375]: 2024-03-24 11:19:04.773351185 +0000 UTC m=+0.199599228 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:19:05.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 podman[61375]: 2024-03-24 11:19:05.023524953 +0000 UTC m=+0.449773001 container create e97233b20e3465a70a732b2c00c6f3d694643c8e37a343771dfcf79aa77fcf78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:19:05.213 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled node-exporter update... 2024-03-24T11:19:05.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 podman[61375]: 2024-03-24 11:19:05.234274111 +0000 UTC m=+0.660522158 container init e97233b20e3465a70a732b2c00c6f3d694643c8e37a343771dfcf79aa77fcf78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-24T11:19:05.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 podman[61375]: 2024-03-24 11:19:05.237350702 +0000 UTC m=+0.663598746 container start e97233b20e3465a70a732b2c00c6f3d694643c8e37a343771dfcf79aa77fcf78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:19:05.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug Started the configuration object watcher 2024-03-24T11:19:05.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug Checking for config object changes every 1s 2024-03-24T11:19:05.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug Processing osd blocklist entries for this node 2024-03-24T11:19:05.747 DEBUG:teuthology.orchestra.run.smithi179:node-exporter.a> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T11:19:05.749 DEBUG:teuthology.orchestra.run.smithi181:node-exporter.b> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T11:19:05.752 INFO:tasks.cephadm:Adding alertmanager.a on smithi179 2024-03-24T11:19:05.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi179=a' 2024-03-24T11:19:06.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug Reading the configuration object to update local LIO configuration 2024-03-24T11:19:06.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug Configuration does not have an entry for this host(smithi181.front.sepia.ceph.com) - nothing to define to LIO 2024-03-24T11:19:06.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-24T11:19:06.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 bash[61375]: e97233b20e3465a70a732b2c00c6f3d694643c8e37a343771dfcf79aa77fcf78 2024-03-24T11:19:06.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:05 smithi181 systemd[1]: Started Ceph iscsi.iscsi.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:19:06.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi179=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:06.316 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: Saving service node-exporter spec with placement smithi179=a;smithi181=b;count:2 2024-03-24T11:19:06.317 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.317 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 6.1 KiB/s wr, 174 op/s 2024-03-24T11:19:06.317 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3670337252' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:19:06.317 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.317 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.317 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.318 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.318 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:06 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:06.471 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:19:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi179=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: Saving service node-exporter spec with placement smithi179=a;smithi181=b;count:2 2024-03-24T11:19:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 6.1 KiB/s wr, 174 op/s 2024-03-24T11:19:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3670337252' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:19:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:06.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi179=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:06.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: Saving service node-exporter spec with placement smithi179=a;smithi181=b;count:2 2024-03-24T11:19:06.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 6.1 KiB/s wr, 174 op/s 2024-03-24T11:19:06.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3670337252' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T11:19:06.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:06.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:06 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:07.590 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:07 smithi181 ceph-mon[33998]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:19:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:07 smithi179 ceph-mon[29608]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:19:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:07 smithi179 ceph-mon[37907]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T11:19:08.206 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled alertmanager update... 2024-03-24T11:19:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:08 smithi181 ceph-mon[33998]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 141 op/s 2024-03-24T11:19:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:08 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:08 smithi181 ceph-mon[33998]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:19:09.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[29608]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 141 op/s 2024-03-24T11:19:09.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:09.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:09.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[29608]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:19:09.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[37907]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.6 KiB/s wr, 141 op/s 2024-03-24T11:19:09.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:09.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:09.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:08 smithi179 ceph-mon[37907]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-24T11:19:09.233 DEBUG:teuthology.orchestra.run.smithi179:alertmanager.a> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T11:19:09.236 INFO:tasks.cephadm:Adding grafana.a on smithi181 2024-03-24T11:19:09.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi181=a' 2024-03-24T11:19:09.832 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:19:09.959 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:09 smithi181 ceph-mon[33998]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:09.959 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:09 smithi181 ceph-mon[33998]: Saving service alertmanager spec with placement smithi179=a;count:1 2024-03-24T11:19:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:09 smithi179 ceph-mon[29608]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:09 smithi179 ceph-mon[29608]: Saving service alertmanager spec with placement smithi179=a;count:1 2024-03-24T11:19:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:09 smithi179 ceph-mon[37907]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:09 smithi179 ceph-mon[37907]: Saving service alertmanager spec with placement smithi179=a;count:1 2024-03-24T11:19:10.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:10 smithi181 ceph-mon[33998]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-03-24T11:19:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:10 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[29608]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[37907]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:11.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:11.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:11.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:11.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:10 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:11.775 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled grafana update... 2024-03-24T11:19:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:11 smithi181 ceph-mon[33998]: Deploying daemon prometheus.a on smithi181 2024-03-24T11:19:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:11 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:12.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:11 smithi179 ceph-mon[29608]: Deploying daemon prometheus.a on smithi181 2024-03-24T11:19:12.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:11 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:11 smithi179 ceph-mon[37907]: Deploying daemon prometheus.a on smithi181 2024-03-24T11:19:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:11 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:12.294 DEBUG:teuthology.orchestra.run.smithi181:grafana.a> sudo journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T11:19:12.297 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:19:12.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:19:12.534 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:12.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[37907]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 101 op/s 2024-03-24T11:19:12.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[37907]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:12.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[37907]: Saving service grafana spec with placement smithi181=a;count:1 2024-03-24T11:19:12.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:12.901 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[29608]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 101 op/s 2024-03-24T11:19:12.901 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[29608]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:12.901 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[29608]: Saving service grafana spec with placement smithi181=a;count:1 2024-03-24T11:19:12.901 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:12 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:13.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:12 smithi181 ceph-mon[33998]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 101 op/s 2024-03-24T11:19:13.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:12 smithi181 ceph-mon[33998]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:12 smithi181 ceph-mon[33998]: Saving service grafana spec with placement smithi181=a;count:1 2024-03-24T11:19:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:12 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:14.508 INFO:teuthology.orchestra.run.smithi179.stdout:[client.0] 2024-03-24T11:19:14.509 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQAyDABmkakdHhAAAdHMcMq+s09wHa0dfo07Vg== 2024-03-24T11:19:15.103 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[37907]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 921 B/s wr, 34 op/s 2024-03-24T11:19:15.104 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/413708770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:15.104 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:15.104 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:19:15.104 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[29608]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 921 B/s wr, 34 op/s 2024-03-24T11:19:15.104 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/413708770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:15.104 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:15.104 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:14 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:19:15.130 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-24T11:19:15.130 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:19:15.130 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:19:15.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:14 smithi181 ceph-mon[33998]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 921 B/s wr, 34 op/s 2024-03-24T11:19:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:14 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/413708770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:14 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:14 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:19:15.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:19:15.407 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:19:15.798 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:19:16.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:15 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:16.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:15 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:15 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:15 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:15 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:16.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:15 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:17.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:16 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:16 smithi181 ceph-mon[33998]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 783 B/s wr, 30 op/s 2024-03-24T11:19:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:16 smithi181 ceph-mon[33998]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:19:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:16 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:16 smithi179 ceph-mon[29608]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 783 B/s wr, 30 op/s 2024-03-24T11:19:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:16 smithi179 ceph-mon[29608]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:19:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:16 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:16 smithi179 ceph-mon[37907]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 783 B/s wr, 30 op/s 2024-03-24T11:19:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:16 smithi179 ceph-mon[37907]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T11:19:18.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-24T11:19:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-24T11:19:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:17 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:19:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-24T11:19:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-24T11:19:18.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:17 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:19:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-24T11:19:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-24T11:19:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:17 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-24T11:19:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:18 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:18 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:18 smithi181 ceph-mon[33998]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-24T11:19:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:18 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-24T11:19:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:18 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-24T11:19:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:18 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:19:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:18 smithi181 ceph-mon[33998]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:19:19.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:19.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:19.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[29608]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-24T11:19:19.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-24T11:19:19.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[29608]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[37907]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-24T11:19:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:18 smithi179 ceph-mon[37907]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T11:19:19.502 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-03-24T11:19:19.503 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQA3DABm1u22HRAA1udzUHauwqTDMp2lDeKjuQ== 2024-03-24T11:19:20.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:19 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:19:20.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:19 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:19:20.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:19 smithi181 ceph-mon[33998]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:19:20.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:19 smithi181 ceph-mon[33998]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:19:20.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.181:0/3219806097' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:20.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:19 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:20.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:19 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:19:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:19:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:19:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[29608]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:19:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[29608]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:19:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.181:0/3219806097' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:19:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:19:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:19:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[37907]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T11:19:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[37907]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T11:19:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.181:0/3219806097' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:19:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:19 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:19:20.819 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-03-24T11:19:20.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T11:19:20.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T11:19:20.863 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:19:20.863 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:19:20.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:19:21.113 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:21.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:20 smithi181 ceph-mon[33998]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 82 B/s, 1 objects/s recovering 2024-03-24T11:19:21.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:20 smithi179 ceph-mon[29608]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 82 B/s, 1 objects/s recovering 2024-03-24T11:19:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:20 smithi179 ceph-mon[37907]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 82 B/s, 1 objects/s recovering 2024-03-24T11:19:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:21 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:21 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:21 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:21 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:22.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:21 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:22.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:21 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:23.081 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:22 smithi181 ceph-mon[33998]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 55 B/s, 0 objects/s recovering 2024-03-24T11:19:23.081 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:22 smithi181 ceph-mon[33998]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:19:23.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:22 smithi179 ceph-mon[29608]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 55 B/s, 0 objects/s recovering 2024-03-24T11:19:23.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:22 smithi179 ceph-mon[29608]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:19:23.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:22 smithi179 ceph-mon[37907]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 55 B/s, 0 objects/s recovering 2024-03-24T11:19:23.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:22 smithi179 ceph-mon[37907]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T11:19:23.335 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:19:23.402 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 podman[62895]: 2024-03-24 11:19:23.314164988 +0000 UTC m=+0.133486622 container create 354037eb0f72829c8ffb7c3136cefc62d10ba3bfb1801092d6602f9d3c9360e1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 podman[62895]: 2024-03-24 11:19:23.50808421 +0000 UTC m=+0.327405852 container init 354037eb0f72829c8ffb7c3136cefc62d10ba3bfb1801092d6602f9d3c9360e1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 podman[62895]: 2024-03-24 11:19:23.510770631 +0000 UTC m=+0.330092276 container start 354037eb0f72829c8ffb7c3136cefc62d10ba3bfb1801092d6602f9d3c9360e1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.534Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.534Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.534Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi181 (none))" 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.534Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.534Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.535Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.536Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:19:23.835 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.537Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.537Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.539Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.539Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.423µs 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.539Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.540Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.540Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=67.21µs wal_replay_duration=662.838µs wbl_replay_duration=122ns total_replay_duration=804.88µs 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.540Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.540Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.540Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.558Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.190496ms db_storage=1.094µs remote_storage=1.144µs web_handler=382ns query_engine=632ns scrape=387.623µs scrape_sd=69.477µs notify=661ns notify_sd=1.378µs rules=16.554414ms tracing=8.882µs 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.558Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:19:23.558Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 bash[62895]: 354037eb0f72829c8ffb7c3136cefc62d10ba3bfb1801092d6602f9d3c9360e1 2024-03-24T11:19:23.836 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:19:23 smithi181 systemd[1]: Started Ceph prometheus.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:19:23.902 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3671971599' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:19:23.902 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:23.902 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:23.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:23.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:19:23.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3671971599' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:19:23.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:23.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:23.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:23.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:23 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:19:23.927 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2865046672},{"type":"v1","addr":"172.21.15.179:6801","nonce":2865046672}]},"active_addr":"172.21.15.179:6801/2865046672","active_change":"2024-03-24T11:14:17.689054+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.179:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":2472348284}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":1519226585}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":4087802319}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":4255117010}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":3185072343}]}]} 2024-03-24T11:19:23.931 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:19:23.932 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:19:23.932 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:19:24.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3671971599' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:19:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' 2024-03-24T11:19:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:23 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:19:24.170 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:24.959 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ignoring --setuser ceph since I am not root 2024-03-24T11:19:24.959 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ignoring --setgroup ceph since I am not root 2024-03-24T11:19:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[29608]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 92 B/s, 2 objects/s recovering 2024-03-24T11:19:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[29608]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:19:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[29608]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:19:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[29608]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:19:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[37907]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 92 B/s, 2 objects/s recovering 2024-03-24T11:19:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[37907]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:19:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[37907]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:19:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:24 smithi179 ceph-mon[37907]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:19:25.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:24 smithi181 ceph-mon[33998]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s; 92 B/s, 2 objects/s recovering 2024-03-24T11:19:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:24 smithi181 ceph-mon[33998]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T11:19:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:24 smithi181 ceph-mon[33998]: from='mgr.14150 172.21.15.179:0/1405643737' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:19:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:24 smithi181 ceph-mon[33998]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T11:19:25.152 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: ignoring --setuser ceph since I am not root 2024-03-24T11:19:25.152 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: ignoring --setgroup ceph since I am not root 2024-03-24T11:19:25.152 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:24 smithi181 ceph-mgr[35734]: -- 172.21.15.181:0/3689948580 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5651155301a0 con 0x565115506800 2024-03-24T11:19:25.311 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:25.216+0000 7f681e4ec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:19:25.651 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:25 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:25.214+0000 7f10f3385200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:19:25.652 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:25 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:25.308+0000 7f10f3385200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:19:25.677 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:25.310+0000 7f681e4ec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:19:26.076 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:19:26.076 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":80,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:13:18.639983+0000","modified":"2024-03-24T11:19:23.986088+0000","last_up_change":"2024-03-24T11:18:40.079955+0000","last_in_change":"2024-03-24T11:18:23.090476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:16:40.018600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T11:18:51.329095+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T11:18:52.018113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:18:53.978924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:18:55.925755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:18:57.991833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14f300f0-7498-4ead-a87e-d9f38bdff21b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6803","nonce":1725503377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6805","nonce":1725503377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6809","nonce":1725503377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6807","nonce":1725503377}]},"public_addr":"172.21.15.179:6803/1725503377","cluster_addr":"172.21.15.179:6805/1725503377","heartbeat_back_addr":"172.21.15.179:6809/1725503377","heartbeat_front_addr":"172.21.15.179:6807/1725503377","state":["exists","up"]},{"osd":1,"uuid":"3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6811","nonce":2564847275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6813","nonce":2564847275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6817","nonce":2564847275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6815","nonce":2564847275}]},"public_addr":"172.21.15.179:6811/2564847275","cluster_addr":"172.21.15.179:6813/2564847275","heartbeat_back_addr":"172.21.15.179:6817/2564847275","heartbeat_front_addr":"172.21.15.179:6815/2564847275","state":["exists","up"]},{"osd":2,"uuid":"863b92b8-c68c-41a3-aed4-fd79e96d34dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6819","nonce":3297713842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6821","nonce":3297713842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6825","nonce":3297713842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6823","nonce":3297713842}]},"public_addr":"172.21.15.179:6819/3297713842","cluster_addr":"172.21.15.179:6821/3297713842","heartbeat_back_addr":"172.21.15.179:6825/3297713842","heartbeat_front_addr":"172.21.15.179:6823/3297713842","state":["exists","up"]},{"osd":3,"uuid":"c3d6b564-195d-4771-9aae-93a9e2eaba52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6827","nonce":944142471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6829","nonce":944142471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6833","nonce":944142471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6831","nonce":944142471}]},"public_addr":"172.21.15.179:6827/944142471","cluster_addr":"172.21.15.179:6829/944142471","heartbeat_back_addr":"172.21.15.179:6833/944142471","heartbeat_front_addr":"172.21.15.179:6831/944142471","state":["exists","up"]},{"osd":4,"uuid":"91bf5303-3a94-407d-aef1-e43bf0e16176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6801","nonce":3231660415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6803","nonce":3231660415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6807","nonce":3231660415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6805","nonce":3231660415}]},"public_addr":"172.21.15.181:6801/3231660415","cluster_addr":"172.21.15.181:6803/3231660415","heartbeat_back_addr":"172.21.15.181:6807/3231660415","heartbeat_front_addr":"172.21.15.181:6805/3231660415","state":["exists","up"]},{"osd":5,"uuid":"cba3f489-e8be-46f7-a281-e69a0ad08646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6809","nonce":775575782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6811","nonce":775575782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6815","nonce":775575782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6813","nonce":775575782}]},"public_addr":"172.21.15.181:6809/775575782","cluster_addr":"172.21.15.181:6811/775575782","heartbeat_back_addr":"172.21.15.181:6815/775575782","heartbeat_front_addr":"172.21.15.181:6813/775575782","state":["exists","up"]},{"osd":6,"uuid":"44a2c3a0-b704-4c10-a257-c54de47ee793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6817","nonce":2357509374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6819","nonce":2357509374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6823","nonce":2357509374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6821","nonce":2357509374}]},"public_addr":"172.21.15.181:6817/2357509374","cluster_addr":"172.21.15.181:6819/2357509374","heartbeat_back_addr":"172.21.15.181:6823/2357509374","heartbeat_front_addr":"172.21.15.181:6821/2357509374","state":["exists","up"]},{"osd":7,"uuid":"df60c4b4-d2f2-4e1f-8e1f-3169c403de00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6825","nonce":732039117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6827","nonce":732039117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6831","nonce":732039117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6829","nonce":732039117}]},"public_addr":"172.21.15.181:6825/732039117","cluster_addr":"172.21.15.181:6827/732039117","heartbeat_back_addr":"172.21.15.181:6831/732039117","heartbeat_front_addr":"172.21.15.181:6829/732039117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:21.394931+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:42.031740+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:53.025982+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:21.055326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:42.268864+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:15.050784+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:17.503900+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:38.510933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2066930919":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/3344305045":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2047439415":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2748363950":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/883221702":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2108875546":"2024-03-25T11:14:17.688945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:19:26.280 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/788365826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:19:26.280 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:25 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:25.949+0000 7f10f3385200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:19:26.280 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.041+0000 7f10f3385200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:19:26.280 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.184+0000 7f10f3385200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:19:26.280 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.279+0000 7f10f3385200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:19:26.339 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/788365826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:19:26.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/788365826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:19:26.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:25.994+0000 7f681e4ec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:19:26.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.091+0000 7f681e4ec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:19:26.340 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.244+0000 7f681e4ec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:19:26.628 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.362+0000 7f10f3385200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:19:26.628 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.628+0000 7f10f3385200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:19:26.692 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.338+0000 7f681e4ec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:19:26.692 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.427+0000 7f681e4ec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:19:26.839 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:19:26.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:19:26.901 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.737+0000 7f10f3385200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:26.901 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.824+0000 7f10f3385200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:19:26.959 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.692+0000 7f681e4ec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:19:26.959 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.803+0000 7f681e4ec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:26.959 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.897+0000 7f681e4ec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:19:27.085 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:27.303 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:26 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:26.995+0000 7f681e4ec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:19:27.303 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:27 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:27.185+0000 7f681e4ec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:19:27.401 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:26 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:26.920+0000 7f10f3385200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:19:27.402 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:27 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:27.097+0000 7f10f3385200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:19:28.401 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:28 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:28.060+0000 7f10f3385200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:19:28.402 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:28 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:28.192+0000 7f10f3385200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:19:28.445 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:28 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:28.170+0000 7f681e4ec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:19:28.446 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:28 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:28.312+0000 7f681e4ec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:19:28.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:28 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:28.660+0000 7f681e4ec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:19:28.816 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:28 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:28.526+0000 7f10f3385200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:19:28.816 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:28 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:28.618+0000 7f10f3385200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:19:28.816 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:28 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:28.815+0000 7f10f3385200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:28.841 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:19:28.841 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":80,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:13:18.639983+0000","modified":"2024-03-24T11:19:23.986088+0000","last_up_change":"2024-03-24T11:18:40.079955+0000","last_in_change":"2024-03-24T11:18:23.090476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:16:40.018600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T11:18:51.329095+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T11:18:52.018113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:18:53.978924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:18:55.925755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:18:57.991833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14f300f0-7498-4ead-a87e-d9f38bdff21b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6803","nonce":1725503377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6805","nonce":1725503377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6809","nonce":1725503377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6807","nonce":1725503377}]},"public_addr":"172.21.15.179:6803/1725503377","cluster_addr":"172.21.15.179:6805/1725503377","heartbeat_back_addr":"172.21.15.179:6809/1725503377","heartbeat_front_addr":"172.21.15.179:6807/1725503377","state":["exists","up"]},{"osd":1,"uuid":"3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6811","nonce":2564847275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6813","nonce":2564847275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6817","nonce":2564847275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6815","nonce":2564847275}]},"public_addr":"172.21.15.179:6811/2564847275","cluster_addr":"172.21.15.179:6813/2564847275","heartbeat_back_addr":"172.21.15.179:6817/2564847275","heartbeat_front_addr":"172.21.15.179:6815/2564847275","state":["exists","up"]},{"osd":2,"uuid":"863b92b8-c68c-41a3-aed4-fd79e96d34dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6819","nonce":3297713842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6821","nonce":3297713842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6825","nonce":3297713842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6823","nonce":3297713842}]},"public_addr":"172.21.15.179:6819/3297713842","cluster_addr":"172.21.15.179:6821/3297713842","heartbeat_back_addr":"172.21.15.179:6825/3297713842","heartbeat_front_addr":"172.21.15.179:6823/3297713842","state":["exists","up"]},{"osd":3,"uuid":"c3d6b564-195d-4771-9aae-93a9e2eaba52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6827","nonce":944142471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6829","nonce":944142471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6833","nonce":944142471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6831","nonce":944142471}]},"public_addr":"172.21.15.179:6827/944142471","cluster_addr":"172.21.15.179:6829/944142471","heartbeat_back_addr":"172.21.15.179:6833/944142471","heartbeat_front_addr":"172.21.15.179:6831/944142471","state":["exists","up"]},{"osd":4,"uuid":"91bf5303-3a94-407d-aef1-e43bf0e16176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6801","nonce":3231660415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6803","nonce":3231660415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6807","nonce":3231660415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6805","nonce":3231660415}]},"public_addr":"172.21.15.181:6801/3231660415","cluster_addr":"172.21.15.181:6803/3231660415","heartbeat_back_addr":"172.21.15.181:6807/3231660415","heartbeat_front_addr":"172.21.15.181:6805/3231660415","state":["exists","up"]},{"osd":5,"uuid":"cba3f489-e8be-46f7-a281-e69a0ad08646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6809","nonce":775575782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6811","nonce":775575782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6815","nonce":775575782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6813","nonce":775575782}]},"public_addr":"172.21.15.181:6809/775575782","cluster_addr":"172.21.15.181:6811/775575782","heartbeat_back_addr":"172.21.15.181:6815/775575782","heartbeat_front_addr":"172.21.15.181:6813/775575782","state":["exists","up"]},{"osd":6,"uuid":"44a2c3a0-b704-4c10-a257-c54de47ee793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6817","nonce":2357509374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6819","nonce":2357509374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6823","nonce":2357509374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6821","nonce":2357509374}]},"public_addr":"172.21.15.181:6817/2357509374","cluster_addr":"172.21.15.181:6819/2357509374","heartbeat_back_addr":"172.21.15.181:6823/2357509374","heartbeat_front_addr":"172.21.15.181:6821/2357509374","state":["exists","up"]},{"osd":7,"uuid":"df60c4b4-d2f2-4e1f-8e1f-3169c403de00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6825","nonce":732039117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6827","nonce":732039117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6831","nonce":732039117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6829","nonce":732039117}]},"public_addr":"172.21.15.181:6825/732039117","cluster_addr":"172.21.15.181:6827/732039117","heartbeat_back_addr":"172.21.15.181:6831/732039117","heartbeat_front_addr":"172.21.15.181:6829/732039117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:21.394931+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:42.031740+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:53.025982+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:21.055326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:42.268864+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:15.050784+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:17.503900+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:18:38.510933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2066930919":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/3344305045":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2047439415":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2748363950":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/883221702":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2108875546":"2024-03-25T11:14:17.688945+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:19:29.151 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3641520586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:19:29.151 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:28 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:28.754+0000 7f681e4ec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:19:29.151 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:28 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:28.961+0000 7f681e4ec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:29.152 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3641520586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:19:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3641520586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:19:29.152 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:29.003+0000 7f10f3385200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:19:29.439 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:29 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:29.150+0000 7f681e4ec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:19:29.548 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:29.287+0000 7f10f3385200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:29.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:19:29.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:19:29.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:19:29.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T11:19:29.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T11:19:29.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T11:19:29.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T11:19:29.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T11:19:29.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:29 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:29.438+0000 7f681e4ec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:29.901 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:29.548+0000 7f10f3385200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:29.901 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:29.787+0000 7f10f3385200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:19:30.091 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.094 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.096 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.100 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.105 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.170 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.176 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.198 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:29 smithi179 ceph-mon[29608]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:19:30.199 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:29 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:29.783+0000 7f681e4ec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:30.199 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:30 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:30.198+0000 7f681e4ec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:19:30.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:29 smithi179 ceph-mon[37907]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:19:30.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:29 smithi181 ceph-mon[33998]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T11:19:30.751 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:30.901 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:30.592+0000 7f10f3385200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:19:31.149 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:30 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:31.149 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:30 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:31.150 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:30 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:31.150 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:30 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:31.150 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:31.038+0000 7f681e4ec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:19:31.286 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:30 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:31.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:30 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:31.287 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:31.177+0000 7f10f3385200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:19:31.620 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:31.286+0000 7f10f3385200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:19:31.620 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:31.500+0000 7f10f3385200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:19:31.620 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: [24/Mar/2024:11:19:31] ENGINE Bus STARTING 2024-03-24T11:19:31.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:31.624+0000 7f681e4ec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:19:31.901 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: CherryPy Checker: 2024-03-24T11:19:31.902 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: The Application mounted at '' has an empty config. 2024-03-24T11:19:31.902 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: 2024-03-24T11:19:31.902 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: [24/Mar/2024:11:19:31] ENGINE Serving on http://:::9283 2024-03-24T11:19:31.902 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x[35730]: [24/Mar/2024:11:19:31] ENGINE Bus STARTED 2024-03-24T11:19:31.997 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:31.724+0000 7f681e4ec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:19:31.997 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:31.948+0000 7f681e4ec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:19:32.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:19:32] ENGINE Bus STARTING 2024-03-24T11:19:32.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: CherryPy Checker: 2024-03-24T11:19:32.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: The Application mounted at '' has an empty config. 2024-03-24T11:19:32.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:19:32.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:19:32] ENGINE Serving on http://:::9283 2024-03-24T11:19:32.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:19:32] ENGINE Bus STARTED 2024-03-24T11:19:32.339 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: Standby manager daemon x restarted 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: Standby manager daemon x started 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: Active manager daemon y restarted 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: Activating manager daemon y 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: mgrmap e18: y(active, starting, since 0.00900239s), standbys: x 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:32.340 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:19:32.341 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: Standby manager daemon x restarted 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: Standby manager daemon x started 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: Active manager daemon y restarted 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: Activating manager daemon y 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: mgrmap e18: y(active, starting, since 0.00900239s), standbys: x 2024-03-24T11:19:32.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:19:32.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:19:32.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:31 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:19:32.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T11:19:32.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: Standby manager daemon x restarted 2024-03-24T11:19:32.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: Standby manager daemon x started 2024-03-24T11:19:32.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.? 172.21.15.181:0/3352769550' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: Active manager daemon y restarted 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: Activating manager daemon y 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: mgrmap e18: y(active, starting, since 0.00900239s), standbys: x 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:19:32.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:19:32.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:31 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:19:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: Manager daemon y is now available 2024-03-24T11:19:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: Manager daemon y is now available 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:19:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:19:33.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:19:33.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: Manager daemon y is now available 2024-03-24T11:19:33.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:33.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:33.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:33.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:19:33.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:19:33.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:19:33.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:19:33.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:19:34.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[29608]: [24/Mar/2024:11:19:33] ENGINE Bus STARTING 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[29608]: mgrmap e19: y(active, since 1.04808s), standbys: x 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[29608]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[29608]: [24/Mar/2024:11:19:33] ENGINE Serving on https://172.21.15.179:7150 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[29608]: [24/Mar/2024:11:19:33] ENGINE Serving on http://172.21.15.179:8765 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[29608]: [24/Mar/2024:11:19:33] ENGINE Bus STARTED 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[37907]: [24/Mar/2024:11:19:33] ENGINE Bus STARTING 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[37907]: mgrmap e19: y(active, since 1.04808s), standbys: x 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[37907]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[37907]: [24/Mar/2024:11:19:33] ENGINE Serving on https://172.21.15.179:7150 2024-03-24T11:19:34.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[37907]: [24/Mar/2024:11:19:33] ENGINE Serving on http://172.21.15.179:8765 2024-03-24T11:19:34.289 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:34 smithi179 ceph-mon[37907]: [24/Mar/2024:11:19:33] ENGINE Bus STARTED 2024-03-24T11:19:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:34 smithi181 ceph-mon[33998]: [24/Mar/2024:11:19:33] ENGINE Bus STARTING 2024-03-24T11:19:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:34 smithi181 ceph-mon[33998]: mgrmap e19: y(active, since 1.04808s), standbys: x 2024-03-24T11:19:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:34 smithi181 ceph-mon[33998]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:34 smithi181 ceph-mon[33998]: [24/Mar/2024:11:19:33] ENGINE Serving on https://172.21.15.179:7150 2024-03-24T11:19:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:34 smithi181 ceph-mon[33998]: [24/Mar/2024:11:19:33] ENGINE Serving on http://172.21.15.179:8765 2024-03-24T11:19:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:34 smithi181 ceph-mon[33998]: [24/Mar/2024:11:19:33] ENGINE Bus STARTED 2024-03-24T11:19:35.093 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:35 smithi181 ceph-mon[33998]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:35.093 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:35 smithi181 ceph-mon[33998]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-24T11:19:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:35 smithi179 ceph-mon[29608]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:35 smithi179 ceph-mon[29608]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-24T11:19:35.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:19:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:19:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:35 smithi179 ceph-mon[37907]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:35.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:35 smithi179 ceph-mon[37907]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-24T11:19:36.496 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:36 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:36.496 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:36 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:36.496 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:36 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:36.496 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:36 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:36.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:36 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:36.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:36 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi181 to 3760M 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi181 to 3760M 2024-03-24T11:19:37.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi181 to 3760M 2024-03-24T11:19:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:19:38.785 INFO:teuthology.orchestra.run.smithi179.stdout:176093659158 2024-03-24T11:19:38.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T11:19:39.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:39 smithi179 ceph-mon[29608]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:39.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:39 smithi179 ceph-mon[37907]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:39.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:39 smithi181 ceph-mon[33998]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-24T11:19:41.374 INFO:teuthology.orchestra.run.smithi179.stdout:253403070477 2024-03-24T11:19:41.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T11:19:41.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:41 smithi179 ceph-mon[37907]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:19:41.510 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:41 smithi179 ceph-mon[29608]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:19:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:41 smithi181 ceph-mon[33998]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T11:19:42.015 INFO:teuthology.orchestra.run.smithi179.stdout:214748364817 2024-03-24T11:19:42.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T11:19:43.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:19:43.500 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:43 smithi179 ceph-mon[29608]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:19:43.500 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:43 smithi179 ceph-mon[37907]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:19:43.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:43 smithi181 ceph-mon[33998]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T11:19:43.895 INFO:teuthology.orchestra.run.smithi179.stdout:55834574892 2024-03-24T11:19:43.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:19:44.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:44.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:44 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:44.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:44 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:44.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:44.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:44 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:44.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:44 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:44.584 INFO:teuthology.orchestra.run.smithi179.stdout:77309411367 2024-03-24T11:19:44.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:19:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:44 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:44 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:45.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:19:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:19:45.362 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:45.384 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:45.401 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:45.488 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:45 smithi179 ceph-mon[37907]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:19:45.489 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:45 smithi179 ceph-mon[37907]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:19:45.489 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:45 smithi179 ceph-mon[29608]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:19:45.489 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:45 smithi179 ceph-mon[29608]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:19:45.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:45 smithi181 ceph-mon[33998]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T11:19:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:45 smithi181 ceph-mon[33998]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T11:19:47.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:47 smithi181 ceph-mon[33998]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:19:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:19:47.653 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:47 smithi179 ceph-mon[29608]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:19:47.653 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:19:47.653 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:47 smithi179 ceph-mon[37907]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:19:47.653 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:19:47.756 INFO:teuthology.orchestra.run.smithi179.stdout:34359738418 2024-03-24T11:19:47.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:19:49.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:49 smithi179 ceph-mon[37907]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:19:49.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:49 smithi179 ceph-mon[37907]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:19:49.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:49 smithi179 ceph-mon[29608]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:19:49.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:49 smithi179 ceph-mon[29608]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:19:49.533 INFO:teuthology.orchestra.run.smithi179.stdout:111669149729 2024-03-24T11:19:49.533 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T11:19:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:49 smithi181 ceph-mon[33998]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:19:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:49 smithi181 ceph-mon[33998]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T11:19:50.376 INFO:teuthology.orchestra.run.smithi179.stdout:176093659161 2024-03-24T11:19:50.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659158 got 176093659161 for osd.5 2024-03-24T11:19:50.959 DEBUG:teuthology.parallel:result is None 2024-03-24T11:19:51.568 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:51 smithi181 ceph-mon[33998]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:19:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2054336675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:19:51.693 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:51 smithi179 ceph-mon[29608]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:19:51.693 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2054336675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:19:51.694 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:51 smithi179 ceph-mon[37907]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:19:51.694 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2054336675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:19:53.316 INFO:teuthology.orchestra.run.smithi179.stdout:77309411369 2024-03-24T11:19:53.370 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:19:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:19:53.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:53 smithi179 ceph-mon[29608]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:19:53.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1800075738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:19:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:53 smithi179 ceph-mon[37907]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:19:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1800075738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:19:53.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:53 smithi181 ceph-mon[33998]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:19:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1800075738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:19:54.242 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-03-24T11:19:54.242 DEBUG:teuthology.parallel:result is None 2024-03-24T11:19:54.465 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:54.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:54.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:19:54.904 INFO:teuthology.orchestra.run.smithi179.stdout:253403070480 2024-03-24T11:19:55.185 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:19:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:19:55.210 INFO:teuthology.orchestra.run.smithi179.stdout:133143986206 2024-03-24T11:19:55.211 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T11:19:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[37907]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:19:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[37907]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1236996998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[29608]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[29608]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:19:55.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:55 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1236996998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:19:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:55 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:55 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:55 smithi181 ceph-mon[33998]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:19:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:55 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:55 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:55 smithi181 ceph-mon[33998]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T11:19:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:55 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1236996998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:19:55.995 INFO:teuthology.orchestra.run.smithi179.stdout:214748364821 2024-03-24T11:19:56.443 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070477 got 253403070480 for osd.7 2024-03-24T11:19:56.443 DEBUG:teuthology.parallel:result is None 2024-03-24T11:19:56.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:56 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:19:56.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:56 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:19:56.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:56 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/626073397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:19:56.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:56 smithi181 ceph-mon[33998]: 3.14 deep-scrub starts 2024-03-24T11:19:56.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:56 smithi181 ceph-mon[33998]: 3.14 deep-scrub ok 2024-03-24T11:19:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:19:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/626073397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[29608]: 3.14 deep-scrub starts 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[29608]: 3.14 deep-scrub ok 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/626073397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[37907]: 3.14 deep-scrub starts 2024-03-24T11:19:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:56 smithi179 ceph-mon[37907]: 3.14 deep-scrub ok 2024-03-24T11:19:57.065 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:57.066 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:57.066 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:57.078 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364817 got 214748364821 for osd.6 2024-03-24T11:19:57.078 DEBUG:teuthology.parallel:result is None 2024-03-24T11:19:57.105 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:57.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:57 smithi179 ceph-mon[29608]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:19:57.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:57 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:19:57.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:57 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:19:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:57 smithi179 ceph-mon[37907]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:19:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:57 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:19:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:57 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:19:57.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:57 smithi181 ceph-mon[33998]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:19:57.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:57 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:19:57.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:57 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:19:58.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:58 smithi181 ceph-mon[33998]: 3.18 scrub starts 2024-03-24T11:19:58.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:58 smithi181 ceph-mon[33998]: 3.18 scrub ok 2024-03-24T11:19:58.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:58 smithi179 ceph-mon[29608]: 3.18 scrub starts 2024-03-24T11:19:58.666 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:58 smithi179 ceph-mon[29608]: 3.18 scrub ok 2024-03-24T11:19:58.666 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:58 smithi179 ceph-mon[37907]: 3.18 scrub starts 2024-03-24T11:19:58.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:58 smithi179 ceph-mon[37907]: 3.18 scrub ok 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: 4.16 deep-scrub starts 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: 4.16 deep-scrub ok 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:19:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:19:59 smithi181 ceph-mon[33998]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:19:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-03-24T11:19:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: 4.16 deep-scrub starts 2024-03-24T11:19:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: 4.16 deep-scrub ok 2024-03-24T11:19:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:19:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:19:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:19:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[29608]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: 4.16 deep-scrub starts 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: 4.16 deep-scrub ok 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:19:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:19:59 smithi179 ceph-mon[37907]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T11:20:00.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:20:00.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:20:00.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:00.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:00.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[29608]: overall HEALTH_OK 2024-03-24T11:20:00.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:20:00.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[37907]: overall HEALTH_OK 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:00 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:20:00.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:00 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:20:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:00 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:20:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:00 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:00 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:00 smithi181 ceph-mon[33998]: overall HEALTH_OK 2024-03-24T11:20:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:00 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:20:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:00 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:20:01.153 INFO:teuthology.orchestra.run.smithi179.stdout:111669149732 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:20:01.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2781255943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:20:01.931 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:01.931 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:20:01.931 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2781255943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.933 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:20:01.934 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:20:01.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2781255943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:20:01.961 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149732 for osd.3 2024-03-24T11:20:01.961 DEBUG:teuthology.parallel:result is None 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi179 to 3760M 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:20:02.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi179 to 3760M 2024-03-24T11:20:02.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:20:02.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:20:02.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:02 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi179 to 3760M 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi181:/etc/ceph/ceph.conf 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: Updating smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:02 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:03.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:20:03.533 INFO:teuthology.orchestra.run.smithi179.stdout:55834574896 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: Deploying daemon alertmanager.a on smithi179 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: 3.8 scrub starts 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: 3.8 scrub ok 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:20:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1627447705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: Deploying daemon alertmanager.a on smithi179 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: 3.8 scrub starts 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: 3.8 scrub ok 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:20:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:03 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1627447705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:20:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: Deploying daemon alertmanager.a on smithi179 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: 3.8 scrub starts 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: 3.8 scrub ok 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:20:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:03 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1627447705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:20:04.189 INFO:teuthology.orchestra.run.smithi179.stdout:34359738421 2024-03-24T11:20:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:20:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:20:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:20:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:20:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3245354379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:20:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:20:04.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:20:04.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:20:04.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:20:04.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:20:04.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:20:04.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:20:04.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3245354379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:20:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:20:04.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:20:04.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3245354379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:20:05.191 INFO:teuthology.orchestra.run.smithi179.stdout:133143986209 2024-03-24T11:20:05.191 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:20:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:20:05.791 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-03-24T11:20:05.791 DEBUG:teuthology.parallel:result is None 2024-03-24T11:20:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:05 smithi181 ceph-mon[33998]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:20:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:05 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:20:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:05 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:20:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:05 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:20:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:05 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:20:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:05 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/372016238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:20:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[29608]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:20:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:20:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:20:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:20:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:20:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/372016238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:20:05.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[37907]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:20:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:20:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:20:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:20:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:20:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:05 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/372016238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:20:06.410 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738421 for osd.0 2024-03-24T11:20:06.410 DEBUG:teuthology.parallel:result is None 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 5.2 deep-scrub starts 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 5.2 deep-scrub ok 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 3.2 scrub starts 2024-03-24T11:20:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 3.2 scrub ok 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 5.2 deep-scrub starts 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 5.2 deep-scrub ok 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 3.2 scrub starts 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 3.2 scrub ok 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:20:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:06 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:20:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 5.2 deep-scrub starts 2024-03-24T11:20:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 5.2 deep-scrub ok 2024-03-24T11:20:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:20:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:20:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:20:06.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:20:06.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 3.2 scrub starts 2024-03-24T11:20:06.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 3.2 scrub ok 2024-03-24T11:20:06.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:20:06.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:06 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:20:07.616 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[29608]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:07.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:20:07.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:20:07.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:20:07.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:20:07.617 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[37907]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:07.617 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:20:07.617 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:20:07.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986209 for osd.4 2024-03-24T11:20:07.824 DEBUG:teuthology.parallel:result is None 2024-03-24T11:20:07.824 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:20:07.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:20:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:07 smithi181 ceph-mon[33998]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:07 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:20:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:07 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:20:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:07 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:20:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:07 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:20:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:20:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:07 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:20:08.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:08 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:20:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:08 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:20:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:08 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:20:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:08 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:20:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:08 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:20:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:08 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:20:08.919 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:20:08.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:20:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:20:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:20:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:20:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:08 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:20:09.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:09 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:20:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:09 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:20:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:09 smithi181 ceph-mon[33998]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:09 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:20:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:09 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[29608]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[37907]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:20:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:09 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:20:10.843 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:10 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:20:10.843 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:10 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:20:10.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:10 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:20:10.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:10 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:20:10.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:10 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:20:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:10 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:20:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:11 smithi179 ceph-mon[29608]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-24T11:20:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:11 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:20:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:11 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:20:11.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:11 smithi179 ceph-mon[37907]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-24T11:20:11.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:11 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:20:11.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:11 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:20:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:11 smithi181 ceph-mon[33998]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-24T11:20:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:11 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:20:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:11 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:20:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:20:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:20:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[29608]: 5.1f deep-scrub starts 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[29608]: 5.1f deep-scrub ok 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[37907]: 5.1f deep-scrub starts 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[37907]: 5.1f deep-scrub ok 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:20:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:12 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:20:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:12 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:20:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:12 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:20:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:12 smithi181 ceph-mon[33998]: 5.1f deep-scrub starts 2024-03-24T11:20:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:12 smithi181 ceph-mon[33998]: 5.1f deep-scrub ok 2024-03-24T11:20:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:12 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:20:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:12 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:20:13.083 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:20:13.087 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:20:13.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:20:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[37907]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:20:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[37907]: 4.1f deep-scrub starts 2024-03-24T11:20:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[37907]: 4.1f deep-scrub ok 2024-03-24T11:20:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:20:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:20:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[29608]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[29608]: 4.1f deep-scrub starts 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[29608]: 4.1f deep-scrub ok 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:20:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:13 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:20:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-mon[33998]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:20:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-mon[33998]: 4.1f deep-scrub starts 2024-03-24T11:20:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-mon[33998]: 4.1f deep-scrub ok 2024-03-24T11:20:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:20:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:20:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:20:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:13 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:20:14.035 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-24T11:20:11.985918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1001,"num_read_kb":847,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249820,"kb_used_data":8732,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711396,"statfs":{"total":767960285184,"available":767704469504,"internally_reserved":0,"allocated":8941568,"data_stored":5089074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996984+0000","last_change":"2024-03-24T11:18:59.704400+0000","last_active":"2024-03-24T11:19:58.996984+0000","last_peered":"2024-03-24T11:19:58.996984+0000","last_clean":"2024-03-24T11:19:58.996984+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:19:58.996984+0000","last_undegraded":"2024-03-24T11:19:58.996984+0000","last_fullsized":"2024-03-24T11:19:58.996984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:14.162529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470398+0000","last_change":"2024-03-24T11:18:57.703289+0000","last_active":"2024-03-24T11:19:59.470398+0000","last_peered":"2024-03-24T11:19:59.470398+0000","last_clean":"2024-03-24T11:19:59.470398+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:19:59.470398+0000","last_undegraded":"2024-03-24T11:19:59.470398+0000","last_fullsized":"2024-03-24T11:19:59.470398+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:14.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470972+0000","last_change":"2024-03-24T11:18:53.684875+0000","last_active":"2024-03-24T11:19:59.470972+0000","last_peered":"2024-03-24T11:19:59.470972+0000","last_clean":"2024-03-24T11:19:59.470972+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:19:59.470972+0000","last_undegraded":"2024-03-24T11:19:59.470972+0000","last_fullsized":"2024-03-24T11:19:59.470972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.844819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996900+0000","last_change":"2024-03-24T11:19:18.824408+0000","last_active":"2024-03-24T11:19:58.996900+0000","last_peered":"2024-03-24T11:19:58.996900+0000","last_clean":"2024-03-24T11:19:58.996900+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:19:58.996900+0000","last_undegraded":"2024-03-24T11:19:58.996900+0000","last_fullsized":"2024-03-24T11:19:58.996900+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.022653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996683+0000","last_change":"2024-03-24T11:19:00.314493+0000","last_active":"2024-03-24T11:19:58.996683+0000","last_peered":"2024-03-24T11:19:58.996683+0000","last_clean":"2024-03-24T11:19:58.996683+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:19:58.996683+0000","last_undegraded":"2024-03-24T11:19:58.996683+0000","last_fullsized":"2024-03-24T11:19:58.996683+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.788301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996925+0000","last_change":"2024-03-24T11:18:57.698383+0000","last_active":"2024-03-24T11:19:58.996925+0000","last_peered":"2024-03-24T11:19:58.996925+0000","last_clean":"2024-03-24T11:19:58.996925+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:19:58.996925+0000","last_undegraded":"2024-03-24T11:19:58.996925+0000","last_fullsized":"2024-03-24T11:19:58.996925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.340946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997539+0000","last_change":"2024-03-24T11:18:53.680266+0000","last_active":"2024-03-24T11:19:58.997539+0000","last_peered":"2024-03-24T11:19:58.997539+0000","last_clean":"2024-03-24T11:19:58.997539+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:19:58.997539+0000","last_undegraded":"2024-03-24T11:19:58.997539+0000","last_fullsized":"2024-03-24T11:19:58.997539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.459989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997007+0000","last_change":"2024-03-24T11:18:55.680984+0000","last_active":"2024-03-24T11:19:58.997007+0000","last_peered":"2024-03-24T11:19:58.997007+0000","last_clean":"2024-03-24T11:19:58.997007+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:19:58.997007+0000","last_undegraded":"2024-03-24T11:19:58.997007+0000","last_fullsized":"2024-03-24T11:19:58.997007+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:24.173013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470777+0000","last_change":"2024-03-24T11:18:59.713465+0000","last_active":"2024-03-24T11:19:59.470777+0000","last_peered":"2024-03-24T11:19:59.470777+0000","last_clean":"2024-03-24T11:19:59.470777+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:19:59.470777+0000","last_undegraded":"2024-03-24T11:19:59.470777+0000","last_fullsized":"2024-03-24T11:19:59.470777+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:01.582786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.382038+0000","last_change":"2024-03-24T11:20:05.382085+0000","last_active":"2024-03-24T11:20:05.382038+0000","last_peered":"2024-03-24T11:20:05.382038+0000","last_clean":"2024-03-24T11:20:05.382038+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:20:05.382038+0000","last_undegraded":"2024-03-24T11:20:05.382038+0000","last_fullsized":"2024-03-24T11:20:05.382038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.381936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.381936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.568574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472614+0000","last_change":"2024-03-24T11:18:53.681043+0000","last_active":"2024-03-24T11:19:59.472614+0000","last_peered":"2024-03-24T11:19:59.472614+0000","last_clean":"2024-03-24T11:19:59.472614+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:19:59.472614+0000","last_undegraded":"2024-03-24T11:19:59.472614+0000","last_fullsized":"2024-03-24T11:19:59.472614+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.521582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997254+0000","last_change":"2024-03-24T11:18:55.682437+0000","last_active":"2024-03-24T11:19:58.997254+0000","last_peered":"2024-03-24T11:19:58.997254+0000","last_clean":"2024-03-24T11:19:58.997254+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:19:58.997254+0000","last_undegraded":"2024-03-24T11:19:58.997254+0000","last_fullsized":"2024-03-24T11:19:58.997254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.808265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470541+0000","last_change":"2024-03-24T11:18:59.707100+0000","last_active":"2024-03-24T11:19:59.470541+0000","last_peered":"2024-03-24T11:19:59.470541+0000","last_clean":"2024-03-24T11:19:59.470541+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:19:59.470541+0000","last_undegraded":"2024-03-24T11:19:59.470541+0000","last_fullsized":"2024-03-24T11:19:59.470541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.566521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470753+0000","last_change":"2024-03-24T11:18:57.701380+0000","last_active":"2024-03-24T11:19:59.470753+0000","last_peered":"2024-03-24T11:19:59.470753+0000","last_clean":"2024-03-24T11:19:59.470753+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:19:59.470753+0000","last_undegraded":"2024-03-24T11:19:59.470753+0000","last_fullsized":"2024-03-24T11:19:59.470753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.709068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471871+0000","last_change":"2024-03-24T11:18:53.681681+0000","last_active":"2024-03-24T11:19:59.471871+0000","last_peered":"2024-03-24T11:19:59.471871+0000","last_clean":"2024-03-24T11:19:59.471871+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:19:59.471871+0000","last_undegraded":"2024-03-24T11:19:59.471871+0000","last_fullsized":"2024-03-24T11:19:59.471871+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.268610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997210+0000","last_change":"2024-03-24T11:18:55.686558+0000","last_active":"2024-03-24T11:19:58.997210+0000","last_peered":"2024-03-24T11:19:58.997210+0000","last_clean":"2024-03-24T11:19:58.997210+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:19:58.997210+0000","last_undegraded":"2024-03-24T11:19:58.997210+0000","last_fullsized":"2024-03-24T11:19:58.997210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.670081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.997154+0000","last_change":"2024-03-24T11:20:00.997176+0000","last_active":"2024-03-24T11:20:00.997154+0000","last_peered":"2024-03-24T11:20:00.997154+0000","last_clean":"2024-03-24T11:20:00.997154+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:20:00.997154+0000","last_undegraded":"2024-03-24T11:20:00.997154+0000","last_fullsized":"2024-03-24T11:20:00.997154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.997096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.997096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.143056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471512+0000","last_change":"2024-03-24T11:18:57.698535+0000","last_active":"2024-03-24T11:19:59.471512+0000","last_peered":"2024-03-24T11:19:59.471512+0000","last_clean":"2024-03-24T11:19:59.471512+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:19:59.471512+0000","last_undegraded":"2024-03-24T11:19:59.471512+0000","last_fullsized":"2024-03-24T11:19:59.471512+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.825235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470544+0000","last_change":"2024-03-24T11:18:53.681845+0000","last_active":"2024-03-24T11:19:59.470544+0000","last_peered":"2024-03-24T11:19:59.470544+0000","last_clean":"2024-03-24T11:19:59.470544+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:19:59.470544+0000","last_undegraded":"2024-03-24T11:19:59.470544+0000","last_fullsized":"2024-03-24T11:19:59.470544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.845615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998063+0000","last_change":"2024-03-24T11:18:55.684816+0000","last_active":"2024-03-24T11:19:58.998063+0000","last_peered":"2024-03-24T11:19:58.998063+0000","last_clean":"2024-03-24T11:19:58.998063+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:19:58.998063+0000","last_undegraded":"2024-03-24T11:19:58.998063+0000","last_fullsized":"2024-03-24T11:19:58.998063+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.595315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996760+0000","last_change":"2024-03-24T11:19:00.312267+0000","last_active":"2024-03-24T11:19:58.996760+0000","last_peered":"2024-03-24T11:19:58.996760+0000","last_clean":"2024-03-24T11:19:58.996760+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:19:58.996760+0000","last_undegraded":"2024-03-24T11:19:58.996760+0000","last_fullsized":"2024-03-24T11:19:58.996760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.807373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997443+0000","last_change":"2024-03-24T11:18:57.699401+0000","last_active":"2024-03-24T11:19:58.997443+0000","last_peered":"2024-03-24T11:19:58.997443+0000","last_clean":"2024-03-24T11:19:58.997443+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:19:58.997443+0000","last_undegraded":"2024-03-24T11:19:58.997443+0000","last_fullsized":"2024-03-24T11:19:58.997443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.033064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997683+0000","last_change":"2024-03-24T11:18:53.672450+0000","last_active":"2024-03-24T11:19:58.997683+0000","last_peered":"2024-03-24T11:19:58.997683+0000","last_clean":"2024-03-24T11:19:58.997683+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:19:58.997683+0000","last_undegraded":"2024-03-24T11:19:58.997683+0000","last_fullsized":"2024-03-24T11:19:58.997683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.274022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996737+0000","last_change":"2024-03-24T11:18:55.684714+0000","last_active":"2024-03-24T11:19:58.996737+0000","last_peered":"2024-03-24T11:19:58.996737+0000","last_clean":"2024-03-24T11:19:58.996737+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:19:58.996737+0000","last_undegraded":"2024-03-24T11:19:58.996737+0000","last_fullsized":"2024-03-24T11:19:58.996737+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.614052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470883+0000","last_change":"2024-03-24T11:18:59.707853+0000","last_active":"2024-03-24T11:19:59.470883+0000","last_peered":"2024-03-24T11:19:59.470883+0000","last_clean":"2024-03-24T11:19:59.470883+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:19:59.470883+0000","last_undegraded":"2024-03-24T11:19:59.470883+0000","last_fullsized":"2024-03-24T11:19:59.470883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.256196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471621+0000","last_change":"2024-03-24T11:18:57.701620+0000","last_active":"2024-03-24T11:19:59.471621+0000","last_peered":"2024-03-24T11:19:59.471621+0000","last_clean":"2024-03-24T11:19:59.471621+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:19:59.471621+0000","last_undegraded":"2024-03-24T11:19:59.471621+0000","last_fullsized":"2024-03-24T11:19:59.471621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.518717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:01.218166+0000","last_change":"2024-03-24T11:20:01.218213+0000","last_active":"2024-03-24T11:20:01.218166+0000","last_peered":"2024-03-24T11:20:01.218166+0000","last_clean":"2024-03-24T11:20:01.218166+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:20:01.218166+0000","last_undegraded":"2024-03-24T11:20:01.218166+0000","last_fullsized":"2024-03-24T11:20:01.218166+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:01.218061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.218061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.318777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.001142+0000","last_change":"2024-03-24T11:20:03.001173+0000","last_active":"2024-03-24T11:20:03.001142+0000","last_peered":"2024-03-24T11:20:03.001142+0000","last_clean":"2024-03-24T11:20:03.001142+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:20:03.001142+0000","last_undegraded":"2024-03-24T11:20:03.001142+0000","last_fullsized":"2024-03-24T11:20:03.001142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:20:03.001079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.001079+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.140666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.208905+0000","last_change":"2024-03-24T11:20:05.208928+0000","last_active":"2024-03-24T11:20:05.208905+0000","last_peered":"2024-03-24T11:20:05.208905+0000","last_clean":"2024-03-24T11:20:05.208905+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:20:05.208905+0000","last_undegraded":"2024-03-24T11:20:05.208905+0000","last_fullsized":"2024-03-24T11:20:05.208905+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.208832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.208832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:28.229456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471236+0000","last_change":"2024-03-24T11:18:57.698222+0000","last_active":"2024-03-24T11:19:59.471236+0000","last_peered":"2024-03-24T11:19:59.471236+0000","last_clean":"2024-03-24T11:19:59.471236+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:19:59.471236+0000","last_undegraded":"2024-03-24T11:19:59.471236+0000","last_fullsized":"2024-03-24T11:19:59.471236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:00.496244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996459+0000","last_change":"2024-03-24T11:19:57.458406+0000","last_active":"2024-03-24T11:19:58.996459+0000","last_peered":"2024-03-24T11:19:58.996459+0000","last_clean":"2024-03-24T11:19:58.996459+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:19:58.996459+0000","last_undegraded":"2024-03-24T11:19:58.996459+0000","last_fullsized":"2024-03-24T11:19:58.996459+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:57.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:57.458291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.184542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471477+0000","last_change":"2024-03-24T11:18:55.687113+0000","last_active":"2024-03-24T11:19:59.471477+0000","last_peered":"2024-03-24T11:19:59.471477+0000","last_clean":"2024-03-24T11:19:59.471477+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:19:59.471477+0000","last_undegraded":"2024-03-24T11:19:59.471477+0000","last_fullsized":"2024-03-24T11:19:59.471477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.674006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.896322+0000","last_change":"2024-03-24T11:20:03.896346+0000","last_active":"2024-03-24T11:20:03.896322+0000","last_peered":"2024-03-24T11:20:03.896322+0000","last_clean":"2024-03-24T11:20:03.896322+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:20:03.896322+0000","last_undegraded":"2024-03-24T11:20:03.896322+0000","last_fullsized":"2024-03-24T11:20:03.896322+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.896225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.896225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.959693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470475+0000","last_change":"2024-03-24T11:18:57.700812+0000","last_active":"2024-03-24T11:19:59.470475+0000","last_peered":"2024-03-24T11:19:59.470475+0000","last_clean":"2024-03-24T11:19:59.470475+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:19:59.470475+0000","last_undegraded":"2024-03-24T11:19:59.470475+0000","last_fullsized":"2024-03-24T11:19:59.470475+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:25.753316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.018180+0000","last_change":"2024-03-24T11:20:07.018200+0000","last_active":"2024-03-24T11:20:07.018180+0000","last_peered":"2024-03-24T11:20:07.018180+0000","last_clean":"2024-03-24T11:20:07.018180+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:20:07.018180+0000","last_undegraded":"2024-03-24T11:20:07.018180+0000","last_fullsized":"2024-03-24T11:20:07.018180+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.018119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.018119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.136141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996480+0000","last_change":"2024-03-24T11:18:55.682010+0000","last_active":"2024-03-24T11:19:58.996480+0000","last_peered":"2024-03-24T11:19:58.996480+0000","last_clean":"2024-03-24T11:19:58.996480+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:19:58.996480+0000","last_undegraded":"2024-03-24T11:19:58.996480+0000","last_fullsized":"2024-03-24T11:19:58.996480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:03.491501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996465+0000","last_change":"2024-03-24T11:19:00.314378+0000","last_active":"2024-03-24T11:19:58.996465+0000","last_peered":"2024-03-24T11:19:58.996465+0000","last_clean":"2024-03-24T11:19:58.996465+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:19:58.996465+0000","last_undegraded":"2024-03-24T11:19:58.996465+0000","last_fullsized":"2024-03-24T11:19:58.996465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.875648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471763+0000","last_change":"2024-03-24T11:18:57.698486+0000","last_active":"2024-03-24T11:19:59.471763+0000","last_peered":"2024-03-24T11:19:59.471763+0000","last_clean":"2024-03-24T11:19:59.471763+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:19:59.471763+0000","last_undegraded":"2024-03-24T11:19:59.471763+0000","last_fullsized":"2024-03-24T11:19:59.471763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.798498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472405+0000","last_change":"2024-03-24T11:18:53.684878+0000","last_active":"2024-03-24T11:19:59.472405+0000","last_peered":"2024-03-24T11:19:59.472405+0000","last_clean":"2024-03-24T11:19:59.472405+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:19:59.472405+0000","last_undegraded":"2024-03-24T11:19:59.472405+0000","last_fullsized":"2024-03-24T11:19:59.472405+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.381938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996821+0000","last_change":"2024-03-24T11:18:56.141810+0000","last_active":"2024-03-24T11:19:58.996821+0000","last_peered":"2024-03-24T11:19:58.996821+0000","last_clean":"2024-03-24T11:19:58.996821+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:19:58.996821+0000","last_undegraded":"2024-03-24T11:19:58.996821+0000","last_fullsized":"2024-03-24T11:19:58.996821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.967564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.469986+0000","last_change":"2024-03-24T11:18:59.714114+0000","last_active":"2024-03-24T11:19:59.469986+0000","last_peered":"2024-03-24T11:19:59.469986+0000","last_clean":"2024-03-24T11:19:59.469986+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:19:59.469986+0000","last_undegraded":"2024-03-24T11:19:59.469986+0000","last_fullsized":"2024-03-24T11:19:59.469986+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.880814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996952+0000","last_change":"2024-03-24T11:18:57.695479+0000","last_active":"2024-03-24T11:19:58.996952+0000","last_peered":"2024-03-24T11:19:58.996952+0000","last_clean":"2024-03-24T11:19:58.996952+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:19:58.996952+0000","last_undegraded":"2024-03-24T11:19:58.996952+0000","last_fullsized":"2024-03-24T11:19:58.996952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.930508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471259+0000","last_change":"2024-03-24T11:18:53.681700+0000","last_active":"2024-03-24T11:19:59.471259+0000","last_peered":"2024-03-24T11:19:59.471259+0000","last_clean":"2024-03-24T11:19:59.471259+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:19:59.471259+0000","last_undegraded":"2024-03-24T11:19:59.471259+0000","last_fullsized":"2024-03-24T11:19:59.471259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.896550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471099+0000","last_change":"2024-03-24T11:18:55.683808+0000","last_active":"2024-03-24T11:19:59.471099+0000","last_peered":"2024-03-24T11:19:59.471099+0000","last_clean":"2024-03-24T11:19:59.471099+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:19:59.471099+0000","last_undegraded":"2024-03-24T11:19:59.471099+0000","last_fullsized":"2024-03-24T11:19:59.471099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.543772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.441848+0000","last_change":"2024-03-24T11:20:03.441868+0000","last_active":"2024-03-24T11:20:03.441848+0000","last_peered":"2024-03-24T11:20:03.441848+0000","last_clean":"2024-03-24T11:20:03.441848+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:20:03.441848+0000","last_undegraded":"2024-03-24T11:20:03.441848+0000","last_fullsized":"2024-03-24T11:20:03.441848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.441791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.441791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.538293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471008+0000","last_change":"2024-03-24T11:18:57.703012+0000","last_active":"2024-03-24T11:19:59.471008+0000","last_peered":"2024-03-24T11:19:59.471008+0000","last_clean":"2024-03-24T11:19:59.471008+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:19:59.471008+0000","last_undegraded":"2024-03-24T11:19:59.471008+0000","last_fullsized":"2024-03-24T11:19:59.471008+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.320906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997581+0000","last_change":"2024-03-24T11:19:56.100757+0000","last_active":"2024-03-24T11:19:58.997581+0000","last_peered":"2024-03-24T11:19:58.997581+0000","last_clean":"2024-03-24T11:19:58.997581+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:19:58.997581+0000","last_undegraded":"2024-03-24T11:19:58.997581+0000","last_fullsized":"2024-03-24T11:19:58.997581+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.100633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.072386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:04.036605+0000","last_change":"2024-03-24T11:20:04.036645+0000","last_active":"2024-03-24T11:20:04.036605+0000","last_peered":"2024-03-24T11:20:04.036605+0000","last_clean":"2024-03-24T11:20:04.036605+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:20:04.036605+0000","last_undegraded":"2024-03-24T11:20:04.036605+0000","last_fullsized":"2024-03-24T11:20:04.036605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:20:04.036510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.036510+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.183695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470693+0000","last_change":"2024-03-24T11:18:59.717539+0000","last_active":"2024-03-24T11:19:59.470693+0000","last_peered":"2024-03-24T11:19:59.470693+0000","last_clean":"2024-03-24T11:19:59.470693+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:19:59.470693+0000","last_undegraded":"2024-03-24T11:19:59.470693+0000","last_fullsized":"2024-03-24T11:19:59.470693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.956051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.390674+0000","last_change":"2024-03-24T11:20:07.390710+0000","last_active":"2024-03-24T11:20:07.390674+0000","last_peered":"2024-03-24T11:20:07.390674+0000","last_clean":"2024-03-24T11:20:07.390674+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:20:07.390674+0000","last_undegraded":"2024-03-24T11:20:07.390674+0000","last_fullsized":"2024-03-24T11:20:07.390674+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:07.390604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.390604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.892391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470985+0000","last_change":"2024-03-24T11:18:53.682165+0000","last_active":"2024-03-24T11:19:59.470985+0000","last_peered":"2024-03-24T11:19:59.470985+0000","last_clean":"2024-03-24T11:19:59.470985+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:19:59.470985+0000","last_undegraded":"2024-03-24T11:19:59.470985+0000","last_fullsized":"2024-03-24T11:19:59.470985+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.317744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997522+0000","last_change":"2024-03-24T11:18:56.141308+0000","last_active":"2024-03-24T11:19:58.997522+0000","last_peered":"2024-03-24T11:19:58.997522+0000","last_clean":"2024-03-24T11:19:58.997522+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:19:58.997522+0000","last_undegraded":"2024-03-24T11:19:58.997522+0000","last_fullsized":"2024-03-24T11:19:58.997522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.469993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998039+0000","last_change":"2024-03-24T11:18:59.705914+0000","last_active":"2024-03-24T11:19:58.998039+0000","last_peered":"2024-03-24T11:19:58.998039+0000","last_clean":"2024-03-24T11:19:58.998039+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:19:58.998039+0000","last_undegraded":"2024-03-24T11:19:58.998039+0000","last_fullsized":"2024-03-24T11:19:58.998039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.058567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:04.436555+0000","last_change":"2024-03-24T11:20:04.436584+0000","last_active":"2024-03-24T11:20:04.436555+0000","last_peered":"2024-03-24T11:20:04.436555+0000","last_clean":"2024-03-24T11:20:04.436555+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:20:04.436555+0000","last_undegraded":"2024-03-24T11:20:04.436555+0000","last_fullsized":"2024-03-24T11:20:04.436555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:04.436493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.436493+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.892034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470274+0000","last_change":"2024-03-24T11:18:53.683017+0000","last_active":"2024-03-24T11:19:59.470274+0000","last_peered":"2024-03-24T11:19:59.470274+0000","last_clean":"2024-03-24T11:19:59.470274+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:19:59.470274+0000","last_undegraded":"2024-03-24T11:19:59.470274+0000","last_fullsized":"2024-03-24T11:19:59.470274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:14.179403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472787+0000","last_change":"2024-03-24T11:18:56.143374+0000","last_active":"2024-03-24T11:19:59.472787+0000","last_peered":"2024-03-24T11:19:59.472787+0000","last_clean":"2024-03-24T11:19:59.472787+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:19:59.472787+0000","last_undegraded":"2024-03-24T11:19:59.472787+0000","last_fullsized":"2024-03-24T11:19:59.472787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.504039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998349+0000","last_change":"2024-03-24T11:18:59.703086+0000","last_active":"2024-03-24T11:19:58.998349+0000","last_peered":"2024-03-24T11:19:58.998349+0000","last_clean":"2024-03-24T11:19:58.998349+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:19:58.998349+0000","last_undegraded":"2024-03-24T11:19:58.998349+0000","last_fullsized":"2024-03-24T11:19:58.998349+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.043850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997187+0000","last_change":"2024-03-24T11:18:57.694184+0000","last_active":"2024-03-24T11:19:58.997187+0000","last_peered":"2024-03-24T11:19:58.997187+0000","last_clean":"2024-03-24T11:19:58.997187+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:19:58.997187+0000","last_undegraded":"2024-03-24T11:19:58.997187+0000","last_fullsized":"2024-03-24T11:19:58.997187+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.074119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472417+0000","last_change":"2024-03-24T11:19:58.799434+0000","last_active":"2024-03-24T11:19:59.472417+0000","last_peered":"2024-03-24T11:19:59.472417+0000","last_clean":"2024-03-24T11:19:59.472417+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:19:59.472417+0000","last_undegraded":"2024-03-24T11:19:59.472417+0000","last_fullsized":"2024-03-24T11:19:59.472417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.799295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.799295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.674050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471090+0000","last_change":"2024-03-24T11:19:58.001386+0000","last_active":"2024-03-24T11:19:59.471090+0000","last_peered":"2024-03-24T11:19:59.471090+0000","last_clean":"2024-03-24T11:19:59.471090+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:19:59.471090+0000","last_undegraded":"2024-03-24T11:19:59.471090+0000","last_fullsized":"2024-03-24T11:19:59.471090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.001296+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.932223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471165+0000","last_change":"2024-03-24T11:19:00.314565+0000","last_active":"2024-03-24T11:19:59.471165+0000","last_peered":"2024-03-24T11:19:59.471165+0000","last_clean":"2024-03-24T11:19:59.471165+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:19:59.471165+0000","last_undegraded":"2024-03-24T11:19:59.471165+0000","last_fullsized":"2024-03-24T11:19:59.471165+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.091963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472513+0000","last_change":"2024-03-24T11:18:57.693993+0000","last_active":"2024-03-24T11:19:59.472513+0000","last_peered":"2024-03-24T11:19:59.472513+0000","last_clean":"2024-03-24T11:19:59.472513+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:19:59.472513+0000","last_undegraded":"2024-03-24T11:19:59.472513+0000","last_fullsized":"2024-03-24T11:19:59.472513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.747158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:01.231673+0000","last_change":"2024-03-24T11:20:01.231696+0000","last_active":"2024-03-24T11:20:01.231673+0000","last_peered":"2024-03-24T11:20:01.231673+0000","last_clean":"2024-03-24T11:20:01.231673+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:20:01.231673+0000","last_undegraded":"2024-03-24T11:20:01.231673+0000","last_fullsized":"2024-03-24T11:20:01.231673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:01.231617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.231617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.010965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997261+0000","last_change":"2024-03-24T11:18:56.142569+0000","last_active":"2024-03-24T11:19:58.997261+0000","last_peered":"2024-03-24T11:19:58.997261+0000","last_clean":"2024-03-24T11:19:58.997261+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:19:58.997261+0000","last_undegraded":"2024-03-24T11:19:58.997261+0000","last_fullsized":"2024-03-24T11:19:58.997261+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.660826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:02.408198+0000","last_change":"2024-03-24T11:20:02.408219+0000","last_active":"2024-03-24T11:20:02.408198+0000","last_peered":"2024-03-24T11:20:02.408198+0000","last_clean":"2024-03-24T11:20:02.408198+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:20:02.408198+0000","last_undegraded":"2024-03-24T11:20:02.408198+0000","last_fullsized":"2024-03-24T11:20:02.408198+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.408130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.408130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:03.752282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471342+0000","last_change":"2024-03-24T11:18:53.684612+0000","last_active":"2024-03-24T11:19:59.471342+0000","last_peered":"2024-03-24T11:19:59.471342+0000","last_clean":"2024-03-24T11:19:59.471342+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:19:59.471342+0000","last_undegraded":"2024-03-24T11:19:59.471342+0000","last_fullsized":"2024-03-24T11:19:59.471342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.251916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470488+0000","last_change":"2024-03-24T11:18:56.142582+0000","last_active":"2024-03-24T11:19:59.470488+0000","last_peered":"2024-03-24T11:19:59.470488+0000","last_clean":"2024-03-24T11:19:59.470488+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:19:59.470488+0000","last_undegraded":"2024-03-24T11:19:59.470488+0000","last_fullsized":"2024-03-24T11:19:59.470488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.969887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.863692+0000","last_change":"2024-03-24T11:20:05.863734+0000","last_active":"2024-03-24T11:20:05.863692+0000","last_peered":"2024-03-24T11:20:05.863692+0000","last_clean":"2024-03-24T11:20:05.863692+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:20:05.863692+0000","last_undegraded":"2024-03-24T11:20:05.863692+0000","last_fullsized":"2024-03-24T11:20:05.863692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:05.863616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.863616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:09.078050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996565+0000","last_change":"2024-03-24T11:18:59.706989+0000","last_active":"2024-03-24T11:19:58.996565+0000","last_peered":"2024-03-24T11:19:58.996565+0000","last_clean":"2024-03-24T11:19:58.996565+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:19:58.996565+0000","last_undegraded":"2024-03-24T11:19:58.996565+0000","last_fullsized":"2024-03-24T11:19:58.996565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.879815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472058+0000","last_change":"2024-03-24T11:18:53.682847+0000","last_active":"2024-03-24T11:19:59.472058+0000","last_peered":"2024-03-24T11:19:59.472058+0000","last_clean":"2024-03-24T11:19:59.472058+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:19:59.472058+0000","last_undegraded":"2024-03-24T11:19:59.472058+0000","last_fullsized":"2024-03-24T11:19:59.472058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.562909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997500+0000","last_change":"2024-03-24T11:18:55.683134+0000","last_active":"2024-03-24T11:19:58.997500+0000","last_peered":"2024-03-24T11:19:58.997500+0000","last_clean":"2024-03-24T11:19:58.997500+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:19:58.997500+0000","last_undegraded":"2024-03-24T11:19:58.997500+0000","last_fullsized":"2024-03-24T11:19:58.997500+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.994319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997439+0000","last_change":"2024-03-24T11:18:57.704315+0000","last_active":"2024-03-24T11:19:58.997439+0000","last_peered":"2024-03-24T11:19:58.997439+0000","last_clean":"2024-03-24T11:19:58.997439+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:19:58.997439+0000","last_undegraded":"2024-03-24T11:19:58.997439+0000","last_fullsized":"2024-03-24T11:19:58.997439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.986639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471181+0000","last_change":"2024-03-24T11:18:59.707271+0000","last_active":"2024-03-24T11:19:59.471181+0000","last_peered":"2024-03-24T11:19:59.471181+0000","last_clean":"2024-03-24T11:19:59.471181+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:19:59.471181+0000","last_undegraded":"2024-03-24T11:19:59.471181+0000","last_fullsized":"2024-03-24T11:19:59.471181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.298674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998236+0000","last_change":"2024-03-24T11:18:57.699926+0000","last_active":"2024-03-24T11:19:58.998236+0000","last_peered":"2024-03-24T11:19:58.998236+0000","last_clean":"2024-03-24T11:19:58.998236+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:19:58.998236+0000","last_undegraded":"2024-03-24T11:19:58.998236+0000","last_fullsized":"2024-03-24T11:19:58.998236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:09.922131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.851778+0000","last_change":"2024-03-24T11:20:00.851820+0000","last_active":"2024-03-24T11:20:00.851778+0000","last_peered":"2024-03-24T11:20:00.851778+0000","last_clean":"2024-03-24T11:20:00.851778+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:20:00.851778+0000","last_undegraded":"2024-03-24T11:20:00.851778+0000","last_fullsized":"2024-03-24T11:20:00.851778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.851691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.851691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.342041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471767+0000","last_change":"2024-03-24T11:18:56.143415+0000","last_active":"2024-03-24T11:19:59.471767+0000","last_peered":"2024-03-24T11:19:59.471767+0000","last_clean":"2024-03-24T11:19:59.471767+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:19:59.471767+0000","last_undegraded":"2024-03-24T11:19:59.471767+0000","last_fullsized":"2024-03-24T11:19:59.471767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.105644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.980171+0000","last_change":"2024-03-24T11:20:07.980193+0000","last_active":"2024-03-24T11:20:07.980171+0000","last_peered":"2024-03-24T11:20:07.980171+0000","last_clean":"2024-03-24T11:20:07.980171+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:20:07.980171+0000","last_undegraded":"2024-03-24T11:20:07.980171+0000","last_fullsized":"2024-03-24T11:20:07.980171+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.980113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.980113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.058894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997984+0000","last_change":"2024-03-24T11:18:57.700062+0000","last_active":"2024-03-24T11:19:58.997984+0000","last_peered":"2024-03-24T11:19:58.997984+0000","last_clean":"2024-03-24T11:19:58.997984+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:19:58.997984+0000","last_undegraded":"2024-03-24T11:19:58.997984+0000","last_fullsized":"2024-03-24T11:19:58.997984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.041178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471983+0000","last_change":"2024-03-24T11:19:56.404250+0000","last_active":"2024-03-24T11:19:59.471983+0000","last_peered":"2024-03-24T11:19:59.471983+0000","last_clean":"2024-03-24T11:19:59.471983+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:19:59.471983+0000","last_undegraded":"2024-03-24T11:19:59.471983+0000","last_fullsized":"2024-03-24T11:19:59.471983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:19:56.404146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.404146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.193173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470638+0000","last_change":"2024-03-24T11:18:55.684166+0000","last_active":"2024-03-24T11:19:59.470638+0000","last_peered":"2024-03-24T11:19:59.470638+0000","last_clean":"2024-03-24T11:19:59.470638+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:19:59.470638+0000","last_undegraded":"2024-03-24T11:19:59.470638+0000","last_fullsized":"2024-03-24T11:19:59.470638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:03.187965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997025+0000","last_change":"2024-03-24T11:18:59.705914+0000","last_active":"2024-03-24T11:19:58.997025+0000","last_peered":"2024-03-24T11:19:58.997025+0000","last_clean":"2024-03-24T11:19:58.997025+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:19:58.997025+0000","last_undegraded":"2024-03-24T11:19:58.997025+0000","last_fullsized":"2024-03-24T11:19:58.997025+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:03.823429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471272+0000","last_change":"2024-03-24T11:18:56.680518+0000","last_active":"2024-03-24T11:19:59.471272+0000","last_peered":"2024-03-24T11:19:59.471272+0000","last_clean":"2024-03-24T11:19:59.471272+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:19:59.471272+0000","last_undegraded":"2024-03-24T11:19:59.471272+0000","last_fullsized":"2024-03-24T11:19:59.471272+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:18:51.653668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.269752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997102+0000","last_change":"2024-03-24T11:18:53.680619+0000","last_active":"2024-03-24T11:19:58.997102+0000","last_peered":"2024-03-24T11:19:58.997102+0000","last_clean":"2024-03-24T11:19:58.997102+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:19:58.997102+0000","last_undegraded":"2024-03-24T11:19:58.997102+0000","last_fullsized":"2024-03-24T11:19:58.997102+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.921367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":120,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471419+0000","last_change":"2024-03-24T11:19:55.086981+0000","last_active":"2024-03-24T11:19:59.471419+0000","last_peered":"2024-03-24T11:19:59.471419+0000","last_clean":"2024-03-24T11:19:59.471419+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:19:59.471419+0000","last_undegraded":"2024-03-24T11:19:59.471419+0000","last_fullsized":"2024-03-24T11:19:59.471419+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:19:55.086847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:19:55.086847+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:14.068671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471091+0000","last_change":"2024-03-24T11:18:57.694386+0000","last_active":"2024-03-24T11:19:59.471091+0000","last_peered":"2024-03-24T11:19:59.471091+0000","last_clean":"2024-03-24T11:19:59.471091+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:19:59.471091+0000","last_undegraded":"2024-03-24T11:19:59.471091+0000","last_fullsized":"2024-03-24T11:19:59.471091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.661605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471639+0000","last_change":"2024-03-24T11:19:00.311903+0000","last_active":"2024-03-24T11:19:59.471639+0000","last_peered":"2024-03-24T11:19:59.471639+0000","last_clean":"2024-03-24T11:19:59.471639+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:19:59.471639+0000","last_undegraded":"2024-03-24T11:19:59.471639+0000","last_fullsized":"2024-03-24T11:19:59.471639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.441322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470446+0000","last_change":"2024-03-24T11:19:58.185937+0000","last_active":"2024-03-24T11:19:59.470446+0000","last_peered":"2024-03-24T11:19:59.470446+0000","last_clean":"2024-03-24T11:19:59.470446+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:19:59.470446+0000","last_undegraded":"2024-03-24T11:19:59.470446+0000","last_fullsized":"2024-03-24T11:19:59.470446+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.185832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.185832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.274032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.555970+0000","last_change":"2024-03-24T11:20:00.555990+0000","last_active":"2024-03-24T11:20:00.555970+0000","last_peered":"2024-03-24T11:20:00.555970+0000","last_clean":"2024-03-24T11:20:00.555970+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:20:00.555970+0000","last_undegraded":"2024-03-24T11:20:00.555970+0000","last_fullsized":"2024-03-24T11:20:00.555970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.555913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.555913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.222390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:02.552300+0000","last_change":"2024-03-24T11:20:02.552338+0000","last_active":"2024-03-24T11:20:02.552300+0000","last_peered":"2024-03-24T11:20:02.552300+0000","last_clean":"2024-03-24T11:20:02.552300+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:20:02.552300+0000","last_undegraded":"2024-03-24T11:20:02.552300+0000","last_fullsized":"2024-03-24T11:20:02.552300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:20:02.552224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.552224+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.289260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997740+0000","last_change":"2024-03-24T11:18:57.702585+0000","last_active":"2024-03-24T11:19:58.997740+0000","last_peered":"2024-03-24T11:19:58.997740+0000","last_clean":"2024-03-24T11:19:58.997740+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:19:58.997740+0000","last_undegraded":"2024-03-24T11:19:58.997740+0000","last_fullsized":"2024-03-24T11:19:58.997740+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.557549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472117+0000","last_change":"2024-03-24T11:18:59.707141+0000","last_active":"2024-03-24T11:19:59.472117+0000","last_peered":"2024-03-24T11:19:59.472117+0000","last_clean":"2024-03-24T11:19:59.472117+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:19:59.472117+0000","last_undegraded":"2024-03-24T11:19:59.472117+0000","last_fullsized":"2024-03-24T11:19:59.472117+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.755888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":306,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471422+0000","last_change":"2024-03-24T11:19:18.724688+0000","last_active":"2024-03-24T11:19:59.471422+0000","last_peered":"2024-03-24T11:19:59.471422+0000","last_clean":"2024-03-24T11:19:59.471422+0000","last_became_active":"2024-03-24T11:18:44.123994+0000","last_became_peered":"2024-03-24T11:18:44.123994+0000","last_unstale":"2024-03-24T11:19:59.471422+0000","last_undegraded":"2024-03-24T11:19:59.471422+0000","last_fullsized":"2024-03-24T11:19:59.471422+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-24T11:19:18.724552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:19:18.724552+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:46.099214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.392249+0000","last_change":"2024-03-24T11:20:05.392270+0000","last_active":"2024-03-24T11:20:05.392249+0000","last_peered":"2024-03-24T11:20:05.392249+0000","last_clean":"2024-03-24T11:20:05.392249+0000","last_became_active":"2024-03-24T11:18:53.678099+0000","last_became_peered":"2024-03-24T11:18:53.678099+0000","last_unstale":"2024-03-24T11:20:05.392249+0000","last_undegraded":"2024-03-24T11:20:05.392249+0000","last_fullsized":"2024-03-24T11:20:05.392249+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.392195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.392195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.644580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":86,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471269+0000","last_change":"2024-03-24T11:18:56.141576+0000","last_active":"2024-03-24T11:19:59.471269+0000","last_peered":"2024-03-24T11:19:59.471269+0000","last_clean":"2024-03-24T11:19:59.471269+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:19:59.471269+0000","last_undegraded":"2024-03-24T11:19:59.471269+0000","last_fullsized":"2024-03-24T11:19:59.471269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.491233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471027+0000","last_change":"2024-03-24T11:18:57.699330+0000","last_active":"2024-03-24T11:19:59.471027+0000","last_peered":"2024-03-24T11:19:59.471027+0000","last_clean":"2024-03-24T11:19:59.471027+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:19:59.471027+0000","last_undegraded":"2024-03-24T11:19:59.471027+0000","last_fullsized":"2024-03-24T11:19:59.471027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.213867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471263+0000","last_change":"2024-03-24T11:18:59.710726+0000","last_active":"2024-03-24T11:19:59.471263+0000","last_peered":"2024-03-24T11:19:59.471263+0000","last_clean":"2024-03-24T11:19:59.471263+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:19:59.471263+0000","last_undegraded":"2024-03-24T11:19:59.471263+0000","last_fullsized":"2024-03-24T11:19:59.471263+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.585199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470639+0000","last_change":"2024-03-24T11:18:53.681515+0000","last_active":"2024-03-24T11:19:59.470639+0000","last_peered":"2024-03-24T11:19:59.470639+0000","last_clean":"2024-03-24T11:19:59.470639+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:19:59.470639+0000","last_undegraded":"2024-03-24T11:19:59.470639+0000","last_fullsized":"2024-03-24T11:19:59.470639+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:01.763672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":141,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:06.880339+0000","last_change":"2024-03-24T11:20:06.880339+0000","last_active":"2024-03-24T11:20:06.880339+0000","last_peered":"2024-03-24T11:20:06.880339+0000","last_clean":"2024-03-24T11:20:06.880339+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:20:06.880339+0000","last_undegraded":"2024-03-24T11:20:06.880339+0000","last_fullsized":"2024-03-24T11:20:06.880339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:20:06.880224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.880224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:32.826706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470771+0000","last_change":"2024-03-24T11:18:55.681576+0000","last_active":"2024-03-24T11:19:59.470771+0000","last_peered":"2024-03-24T11:19:59.470771+0000","last_clean":"2024-03-24T11:19:59.470771+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:19:59.470771+0000","last_undegraded":"2024-03-24T11:19:59.470771+0000","last_fullsized":"2024-03-24T11:19:59.470771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.955414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472509+0000","last_change":"2024-03-24T11:18:57.693189+0000","last_active":"2024-03-24T11:19:59.472509+0000","last_peered":"2024-03-24T11:19:59.472509+0000","last_clean":"2024-03-24T11:19:59.472509+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:19:59.472509+0000","last_undegraded":"2024-03-24T11:19:59.472509+0000","last_fullsized":"2024-03-24T11:19:59.472509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.238110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470286+0000","last_change":"2024-03-24T11:19:00.312423+0000","last_active":"2024-03-24T11:19:59.470286+0000","last_peered":"2024-03-24T11:19:59.470286+0000","last_clean":"2024-03-24T11:19:59.470286+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:19:59.470286+0000","last_undegraded":"2024-03-24T11:19:59.470286+0000","last_fullsized":"2024-03-24T11:19:59.470286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:11.289224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.202646+0000","last_change":"2024-03-24T11:20:07.202667+0000","last_active":"2024-03-24T11:20:07.202646+0000","last_peered":"2024-03-24T11:20:07.202646+0000","last_clean":"2024-03-24T11:20:07.202646+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:20:07.202646+0000","last_undegraded":"2024-03-24T11:20:07.202646+0000","last_fullsized":"2024-03-24T11:20:07.202646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.202576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.202576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:10.625674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470502+0000","last_change":"2024-03-24T11:18:56.142025+0000","last_active":"2024-03-24T11:19:59.470502+0000","last_peered":"2024-03-24T11:19:59.470502+0000","last_clean":"2024-03-24T11:19:59.470502+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:19:59.470502+0000","last_undegraded":"2024-03-24T11:19:59.470502+0000","last_fullsized":"2024-03-24T11:19:59.470502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.374097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.186154+0000","last_change":"2024-03-24T11:20:05.186178+0000","last_active":"2024-03-24T11:20:05.186154+0000","last_peered":"2024-03-24T11:20:05.186154+0000","last_clean":"2024-03-24T11:20:05.186154+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:20:05.186154+0000","last_undegraded":"2024-03-24T11:20:05.186154+0000","last_fullsized":"2024-03-24T11:20:05.186154+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.186078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.695962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.469989+0000","last_change":"2024-03-24T11:18:59.714865+0000","last_active":"2024-03-24T11:19:59.469989+0000","last_peered":"2024-03-24T11:19:59.469989+0000","last_clean":"2024-03-24T11:19:59.469989+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:19:59.469989+0000","last_undegraded":"2024-03-24T11:19:59.469989+0000","last_fullsized":"2024-03-24T11:19:59.469989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.985683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472708+0000","last_change":"2024-03-24T11:18:53.681275+0000","last_active":"2024-03-24T11:19:59.472708+0000","last_peered":"2024-03-24T11:19:59.472708+0000","last_clean":"2024-03-24T11:19:59.472708+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:19:59.472708+0000","last_undegraded":"2024-03-24T11:19:59.472708+0000","last_fullsized":"2024-03-24T11:19:59.472708+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.449318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471059+0000","last_change":"2024-03-24T11:18:55.687126+0000","last_active":"2024-03-24T11:19:59.471059+0000","last_peered":"2024-03-24T11:19:59.471059+0000","last_clean":"2024-03-24T11:19:59.471059+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:19:59.471059+0000","last_undegraded":"2024-03-24T11:19:59.471059+0000","last_fullsized":"2024-03-24T11:19:59.471059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.723563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:01.039976+0000","last_change":"2024-03-24T11:20:01.039976+0000","last_active":"2024-03-24T11:20:01.039976+0000","last_peered":"2024-03-24T11:20:01.039976+0000","last_clean":"2024-03-24T11:20:01.039976+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:20:01.039976+0000","last_undegraded":"2024-03-24T11:20:01.039976+0000","last_fullsized":"2024-03-24T11:20:01.039976+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:01.039839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.039839+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:01.168501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471214+0000","last_change":"2024-03-24T11:19:00.314552+0000","last_active":"2024-03-24T11:19:59.471214+0000","last_peered":"2024-03-24T11:19:59.471214+0000","last_clean":"2024-03-24T11:19:59.471214+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:19:59.471214+0000","last_undegraded":"2024-03-24T11:19:59.471214+0000","last_fullsized":"2024-03-24T11:19:59.471214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.506599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":54,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470940+0000","last_change":"2024-03-24T11:18:53.683466+0000","last_active":"2024-03-24T11:19:59.470940+0000","last_peered":"2024-03-24T11:19:59.470940+0000","last_clean":"2024-03-24T11:19:59.470940+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:19:59.470940+0000","last_undegraded":"2024-03-24T11:19:59.470940+0000","last_fullsized":"2024-03-24T11:19:59.470940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.603409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997139+0000","last_change":"2024-03-24T11:18:55.681171+0000","last_active":"2024-03-24T11:19:58.997139+0000","last_peered":"2024-03-24T11:19:58.997139+0000","last_clean":"2024-03-24T11:19:58.997139+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:19:58.997139+0000","last_undegraded":"2024-03-24T11:19:58.997139+0000","last_fullsized":"2024-03-24T11:19:58.997139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.814781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997229+0000","last_change":"2024-03-24T11:18:57.697565+0000","last_active":"2024-03-24T11:19:58.997229+0000","last_peered":"2024-03-24T11:19:58.997229+0000","last_clean":"2024-03-24T11:19:58.997229+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:19:58.997229+0000","last_undegraded":"2024-03-24T11:19:58.997229+0000","last_fullsized":"2024-03-24T11:19:58.997229+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.519845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997134+0000","last_change":"2024-03-24T11:18:59.706057+0000","last_active":"2024-03-24T11:19:58.997134+0000","last_peered":"2024-03-24T11:19:58.997134+0000","last_clean":"2024-03-24T11:19:58.997134+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:19:58.997134+0000","last_undegraded":"2024-03-24T11:19:58.997134+0000","last_fullsized":"2024-03-24T11:19:58.997134+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.016361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997004+0000","last_change":"2024-03-24T11:18:53.679203+0000","last_active":"2024-03-24T11:19:58.997004+0000","last_peered":"2024-03-24T11:19:58.997004+0000","last_clean":"2024-03-24T11:19:58.997004+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:19:58.997004+0000","last_undegraded":"2024-03-24T11:19:58.997004+0000","last_fullsized":"2024-03-24T11:19:58.997004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.063996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997088+0000","last_change":"2024-03-24T11:18:56.141573+0000","last_active":"2024-03-24T11:19:58.997088+0000","last_peered":"2024-03-24T11:19:58.997088+0000","last_clean":"2024-03-24T11:19:58.997088+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:19:58.997088+0000","last_undegraded":"2024-03-24T11:19:58.997088+0000","last_fullsized":"2024-03-24T11:19:58.997088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.850624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.029170+0000","last_change":"2024-03-24T11:20:00.029193+0000","last_active":"2024-03-24T11:20:00.029170+0000","last_peered":"2024-03-24T11:20:00.029170+0000","last_clean":"2024-03-24T11:20:00.029170+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:20:00.029170+0000","last_undegraded":"2024-03-24T11:20:00.029170+0000","last_fullsized":"2024-03-24T11:20:00.029170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.029091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.029091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.480294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472484+0000","last_change":"2024-03-24T11:18:59.715347+0000","last_active":"2024-03-24T11:19:59.472484+0000","last_peered":"2024-03-24T11:19:59.472484+0000","last_clean":"2024-03-24T11:19:59.472484+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:19:59.472484+0000","last_undegraded":"2024-03-24T11:19:59.472484+0000","last_fullsized":"2024-03-24T11:19:59.472484+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.582227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997908+0000","last_change":"2024-03-24T11:18:57.699380+0000","last_active":"2024-03-24T11:19:58.997908+0000","last_peered":"2024-03-24T11:19:58.997908+0000","last_clean":"2024-03-24T11:19:58.997908+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:19:58.997908+0000","last_undegraded":"2024-03-24T11:19:58.997908+0000","last_fullsized":"2024-03-24T11:19:58.997908+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.138955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:02.414959+0000","last_change":"2024-03-24T11:20:02.414997+0000","last_active":"2024-03-24T11:20:02.414959+0000","last_peered":"2024-03-24T11:20:02.414959+0000","last_clean":"2024-03-24T11:20:02.414959+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:20:02.414959+0000","last_undegraded":"2024-03-24T11:20:02.414959+0000","last_fullsized":"2024-03-24T11:20:02.414959+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.414845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.414845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.613098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471783+0000","last_change":"2024-03-24T11:18:55.686410+0000","last_active":"2024-03-24T11:19:59.471783+0000","last_peered":"2024-03-24T11:19:59.471783+0000","last_clean":"2024-03-24T11:19:59.471783+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:19:59.471783+0000","last_undegraded":"2024-03-24T11:19:59.471783+0000","last_fullsized":"2024-03-24T11:19:59.471783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997319+0000","last_change":"2024-03-24T11:19:00.312496+0000","last_active":"2024-03-24T11:19:58.997319+0000","last_peered":"2024-03-24T11:19:58.997319+0000","last_clean":"2024-03-24T11:19:58.997319+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:19:58.997319+0000","last_undegraded":"2024-03-24T11:19:58.997319+0000","last_fullsized":"2024-03-24T11:19:58.997319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.512245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472481+0000","last_change":"2024-03-24T11:18:57.696783+0000","last_active":"2024-03-24T11:19:59.472481+0000","last_peered":"2024-03-24T11:19:59.472481+0000","last_clean":"2024-03-24T11:19:59.472481+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:19:59.472481+0000","last_undegraded":"2024-03-24T11:19:59.472481+0000","last_fullsized":"2024-03-24T11:19:59.472481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.106375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996975+0000","last_change":"2024-03-24T11:18:53.679465+0000","last_active":"2024-03-24T11:19:58.996975+0000","last_peered":"2024-03-24T11:19:58.996975+0000","last_clean":"2024-03-24T11:19:58.996975+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:19:58.996975+0000","last_undegraded":"2024-03-24T11:19:58.996975+0000","last_fullsized":"2024-03-24T11:19:58.996975+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.456813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997636+0000","last_change":"2024-03-24T11:18:55.684671+0000","last_active":"2024-03-24T11:19:58.997636+0000","last_peered":"2024-03-24T11:19:58.997636+0000","last_clean":"2024-03-24T11:19:58.997636+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:19:58.997636+0000","last_undegraded":"2024-03-24T11:19:58.997636+0000","last_fullsized":"2024-03-24T11:19:58.997636+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.694354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:06.548093+0000","last_change":"2024-03-24T11:20:06.548123+0000","last_active":"2024-03-24T11:20:06.548093+0000","last_peered":"2024-03-24T11:20:06.548093+0000","last_clean":"2024-03-24T11:20:06.548093+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:20:06.548093+0000","last_undegraded":"2024-03-24T11:20:06.548093+0000","last_fullsized":"2024-03-24T11:20:06.548093+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:06.548036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.548036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:08.537924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:04.159001+0000","last_change":"2024-03-24T11:20:04.159022+0000","last_active":"2024-03-24T11:20:04.159001+0000","last_peered":"2024-03-24T11:20:04.159001+0000","last_clean":"2024-03-24T11:20:04.159001+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:20:04.159001+0000","last_undegraded":"2024-03-24T11:20:04.159001+0000","last_fullsized":"2024-03-24T11:20:04.159001+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:04.158945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.158945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.212380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.233296+0000","last_change":"2024-03-24T11:20:03.233345+0000","last_active":"2024-03-24T11:20:03.233296+0000","last_peered":"2024-03-24T11:20:03.233296+0000","last_clean":"2024-03-24T11:20:03.233296+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:20:03.233296+0000","last_undegraded":"2024-03-24T11:20:03.233296+0000","last_fullsized":"2024-03-24T11:20:03.233296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:03.233200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.233200+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.734595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997689+0000","last_change":"2024-03-24T11:19:18.829942+0000","last_active":"2024-03-24T11:19:58.997689+0000","last_peered":"2024-03-24T11:19:58.997689+0000","last_clean":"2024-03-24T11:19:58.997689+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:19:58.997689+0000","last_undegraded":"2024-03-24T11:19:58.997689+0000","last_fullsized":"2024-03-24T11:19:58.997689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.426875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997890+0000","last_change":"2024-03-24T11:18:59.703044+0000","last_active":"2024-03-24T11:19:58.997890+0000","last_peered":"2024-03-24T11:19:58.997890+0000","last_clean":"2024-03-24T11:19:58.997890+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:19:58.997890+0000","last_undegraded":"2024-03-24T11:19:58.997890+0000","last_fullsized":"2024-03-24T11:19:58.997890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.281189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998374+0000","last_change":"2024-03-24T11:18:57.701581+0000","last_active":"2024-03-24T11:19:58.998374+0000","last_peered":"2024-03-24T11:19:58.998374+0000","last_clean":"2024-03-24T11:19:58.998374+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:19:58.998374+0000","last_undegraded":"2024-03-24T11:19:58.998374+0000","last_fullsized":"2024-03-24T11:19:58.998374+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.273789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997367+0000","last_change":"2024-03-24T11:18:53.685545+0000","last_active":"2024-03-24T11:19:58.997367+0000","last_peered":"2024-03-24T11:19:58.997367+0000","last_clean":"2024-03-24T11:19:58.997367+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:19:58.997367+0000","last_undegraded":"2024-03-24T11:19:58.997367+0000","last_fullsized":"2024-03-24T11:19:58.997367+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.153480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997299+0000","last_change":"2024-03-24T11:18:55.680886+0000","last_active":"2024-03-24T11:19:58.997299+0000","last_peered":"2024-03-24T11:19:58.997299+0000","last_clean":"2024-03-24T11:19:58.997299+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:19:58.997299+0000","last_undegraded":"2024-03-24T11:19:58.997299+0000","last_fullsized":"2024-03-24T11:19:58.997299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.995747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":397,"ondisk_log_size":397,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070483,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32624,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712528,"statfs":{"total":95995035648,"available":95961628672,"internally_reserved":0,"allocated":1482752,"data_stored":1004988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364824,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32612,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712540,"statfs":{"total":95995035648,"available":95961640960,"internally_reserved":0,"allocated":1470464,"data_stored":1005209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":415678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32632,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712520,"statfs":{"total":95995035648,"available":95961620480,"internally_reserved":0,"allocated":1490944,"data_stored":1004743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":414136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":414870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":901120,"data_stored":415048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659165,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":414402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:20:14.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:20:14.748 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:14.877 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:20:14 smithi179 systemd[1]: Started Ceph alertmanager.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 4.10 deep-scrub starts 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[29608]: 4.10 deep-scrub ok 2024-03-24T11:20:14.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 4.10 deep-scrub starts 2024-03-24T11:20:14.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-mon[37907]: 4.10 deep-scrub ok 2024-03-24T11:20:14.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:14.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:14.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:20:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:20:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:20:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:20:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:20:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:20:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 4.10 deep-scrub starts 2024-03-24T11:20:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:14 smithi181 ceph-mon[33998]: 4.10 deep-scrub ok 2024-03-24T11:20:15.138 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:14] ENGINE Bus STOPPING 2024-03-24T11:20:15.138 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:20:15.138 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:14] ENGINE Bus STOPPED 2024-03-24T11:20:15.138 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:14] ENGINE Bus STARTING 2024-03-24T11:20:15.138 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:15] ENGINE Serving on http://:::9283 2024-03-24T11:20:15.139 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:15] ENGINE Bus STARTED 2024-03-24T11:20:15.139 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:20:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:20:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:20:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:20:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:20:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:16.707 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:20:16.709 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:20:16.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: Deploying daemon grafana.a on smithi181 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 6.e deep-scrub starts 2024-03-24T11:20:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[37907]: 6.e deep-scrub ok 2024-03-24T11:20:16.960 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[72689]: ts=2024-03-24T11:20:16.585Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000161465s 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: Deploying daemon grafana.a on smithi181 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 6.e deep-scrub starts 2024-03-24T11:20:16.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:16 smithi179 ceph-mon[29608]: 6.e deep-scrub ok 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: Deploying daemon grafana.a on smithi181 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:20:17.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:20:17.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:20:17.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 6.e deep-scrub starts 2024-03-24T11:20:17.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:16 smithi181 ceph-mon[33998]: 6.e deep-scrub ok 2024-03-24T11:20:17.739 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-24T11:20:15.987043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1006,"num_read_kb":852,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":610,"ondisk_log_size":610,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249820,"kb_used_data":8732,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711396,"statfs":{"total":767960285184,"available":767704469504,"internally_reserved":0,"allocated":8941568,"data_stored":5089074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996984+0000","last_change":"2024-03-24T11:18:59.704400+0000","last_active":"2024-03-24T11:19:58.996984+0000","last_peered":"2024-03-24T11:19:58.996984+0000","last_clean":"2024-03-24T11:19:58.996984+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:19:58.996984+0000","last_undegraded":"2024-03-24T11:19:58.996984+0000","last_fullsized":"2024-03-24T11:19:58.996984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:14.162529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470398+0000","last_change":"2024-03-24T11:18:57.703289+0000","last_active":"2024-03-24T11:19:59.470398+0000","last_peered":"2024-03-24T11:19:59.470398+0000","last_clean":"2024-03-24T11:19:59.470398+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:19:59.470398+0000","last_undegraded":"2024-03-24T11:19:59.470398+0000","last_fullsized":"2024-03-24T11:19:59.470398+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:14.786248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470972+0000","last_change":"2024-03-24T11:18:53.684875+0000","last_active":"2024-03-24T11:19:59.470972+0000","last_peered":"2024-03-24T11:19:59.470972+0000","last_clean":"2024-03-24T11:19:59.470972+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:19:59.470972+0000","last_undegraded":"2024-03-24T11:19:59.470972+0000","last_fullsized":"2024-03-24T11:19:59.470972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.844819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996900+0000","last_change":"2024-03-24T11:19:18.824408+0000","last_active":"2024-03-24T11:19:58.996900+0000","last_peered":"2024-03-24T11:19:58.996900+0000","last_clean":"2024-03-24T11:19:58.996900+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:19:58.996900+0000","last_undegraded":"2024-03-24T11:19:58.996900+0000","last_fullsized":"2024-03-24T11:19:58.996900+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.022653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996683+0000","last_change":"2024-03-24T11:19:00.314493+0000","last_active":"2024-03-24T11:19:58.996683+0000","last_peered":"2024-03-24T11:19:58.996683+0000","last_clean":"2024-03-24T11:19:58.996683+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:19:58.996683+0000","last_undegraded":"2024-03-24T11:19:58.996683+0000","last_fullsized":"2024-03-24T11:19:58.996683+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.788301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996925+0000","last_change":"2024-03-24T11:18:57.698383+0000","last_active":"2024-03-24T11:19:58.996925+0000","last_peered":"2024-03-24T11:19:58.996925+0000","last_clean":"2024-03-24T11:19:58.996925+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:19:58.996925+0000","last_undegraded":"2024-03-24T11:19:58.996925+0000","last_fullsized":"2024-03-24T11:19:58.996925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.340946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997539+0000","last_change":"2024-03-24T11:18:53.680266+0000","last_active":"2024-03-24T11:19:58.997539+0000","last_peered":"2024-03-24T11:19:58.997539+0000","last_clean":"2024-03-24T11:19:58.997539+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:19:58.997539+0000","last_undegraded":"2024-03-24T11:19:58.997539+0000","last_fullsized":"2024-03-24T11:19:58.997539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.459989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997007+0000","last_change":"2024-03-24T11:18:55.680984+0000","last_active":"2024-03-24T11:19:58.997007+0000","last_peered":"2024-03-24T11:19:58.997007+0000","last_clean":"2024-03-24T11:19:58.997007+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:19:58.997007+0000","last_undegraded":"2024-03-24T11:19:58.997007+0000","last_fullsized":"2024-03-24T11:19:58.997007+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:24.173013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470777+0000","last_change":"2024-03-24T11:18:59.713465+0000","last_active":"2024-03-24T11:19:59.470777+0000","last_peered":"2024-03-24T11:19:59.470777+0000","last_clean":"2024-03-24T11:19:59.470777+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:19:59.470777+0000","last_undegraded":"2024-03-24T11:19:59.470777+0000","last_fullsized":"2024-03-24T11:19:59.470777+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:01.582786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.382038+0000","last_change":"2024-03-24T11:20:05.382085+0000","last_active":"2024-03-24T11:20:05.382038+0000","last_peered":"2024-03-24T11:20:05.382038+0000","last_clean":"2024-03-24T11:20:05.382038+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:20:05.382038+0000","last_undegraded":"2024-03-24T11:20:05.382038+0000","last_fullsized":"2024-03-24T11:20:05.382038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.381936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.381936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.568574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472614+0000","last_change":"2024-03-24T11:18:53.681043+0000","last_active":"2024-03-24T11:19:59.472614+0000","last_peered":"2024-03-24T11:19:59.472614+0000","last_clean":"2024-03-24T11:19:59.472614+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:19:59.472614+0000","last_undegraded":"2024-03-24T11:19:59.472614+0000","last_fullsized":"2024-03-24T11:19:59.472614+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.521582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997254+0000","last_change":"2024-03-24T11:18:55.682437+0000","last_active":"2024-03-24T11:19:58.997254+0000","last_peered":"2024-03-24T11:19:58.997254+0000","last_clean":"2024-03-24T11:19:58.997254+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:19:58.997254+0000","last_undegraded":"2024-03-24T11:19:58.997254+0000","last_fullsized":"2024-03-24T11:19:58.997254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.808265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470541+0000","last_change":"2024-03-24T11:18:59.707100+0000","last_active":"2024-03-24T11:19:59.470541+0000","last_peered":"2024-03-24T11:19:59.470541+0000","last_clean":"2024-03-24T11:19:59.470541+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:19:59.470541+0000","last_undegraded":"2024-03-24T11:19:59.470541+0000","last_fullsized":"2024-03-24T11:19:59.470541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.566521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470753+0000","last_change":"2024-03-24T11:18:57.701380+0000","last_active":"2024-03-24T11:19:59.470753+0000","last_peered":"2024-03-24T11:19:59.470753+0000","last_clean":"2024-03-24T11:19:59.470753+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:19:59.470753+0000","last_undegraded":"2024-03-24T11:19:59.470753+0000","last_fullsized":"2024-03-24T11:19:59.470753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.709068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471871+0000","last_change":"2024-03-24T11:18:53.681681+0000","last_active":"2024-03-24T11:19:59.471871+0000","last_peered":"2024-03-24T11:19:59.471871+0000","last_clean":"2024-03-24T11:19:59.471871+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:19:59.471871+0000","last_undegraded":"2024-03-24T11:19:59.471871+0000","last_fullsized":"2024-03-24T11:19:59.471871+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.268610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997210+0000","last_change":"2024-03-24T11:18:55.686558+0000","last_active":"2024-03-24T11:19:58.997210+0000","last_peered":"2024-03-24T11:19:58.997210+0000","last_clean":"2024-03-24T11:19:58.997210+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:19:58.997210+0000","last_undegraded":"2024-03-24T11:19:58.997210+0000","last_fullsized":"2024-03-24T11:19:58.997210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.670081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.997154+0000","last_change":"2024-03-24T11:20:00.997176+0000","last_active":"2024-03-24T11:20:00.997154+0000","last_peered":"2024-03-24T11:20:00.997154+0000","last_clean":"2024-03-24T11:20:00.997154+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:20:00.997154+0000","last_undegraded":"2024-03-24T11:20:00.997154+0000","last_fullsized":"2024-03-24T11:20:00.997154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.997096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.997096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.143056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471512+0000","last_change":"2024-03-24T11:18:57.698535+0000","last_active":"2024-03-24T11:19:59.471512+0000","last_peered":"2024-03-24T11:19:59.471512+0000","last_clean":"2024-03-24T11:19:59.471512+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:19:59.471512+0000","last_undegraded":"2024-03-24T11:19:59.471512+0000","last_fullsized":"2024-03-24T11:19:59.471512+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.825235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470544+0000","last_change":"2024-03-24T11:18:53.681845+0000","last_active":"2024-03-24T11:19:59.470544+0000","last_peered":"2024-03-24T11:19:59.470544+0000","last_clean":"2024-03-24T11:19:59.470544+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:19:59.470544+0000","last_undegraded":"2024-03-24T11:19:59.470544+0000","last_fullsized":"2024-03-24T11:19:59.470544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.845615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:11.516095+0000","last_change":"2024-03-24T11:20:11.516126+0000","last_active":"2024-03-24T11:20:11.516095+0000","last_peered":"2024-03-24T11:20:11.516095+0000","last_clean":"2024-03-24T11:20:11.516095+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:20:11.516095+0000","last_undegraded":"2024-03-24T11:20:11.516095+0000","last_fullsized":"2024-03-24T11:20:11.516095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:20:11.516026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:11.516026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.641243+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996760+0000","last_change":"2024-03-24T11:19:00.312267+0000","last_active":"2024-03-24T11:19:58.996760+0000","last_peered":"2024-03-24T11:19:58.996760+0000","last_clean":"2024-03-24T11:19:58.996760+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:19:58.996760+0000","last_undegraded":"2024-03-24T11:19:58.996760+0000","last_fullsized":"2024-03-24T11:19:58.996760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.807373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997443+0000","last_change":"2024-03-24T11:18:57.699401+0000","last_active":"2024-03-24T11:19:58.997443+0000","last_peered":"2024-03-24T11:19:58.997443+0000","last_clean":"2024-03-24T11:19:58.997443+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:19:58.997443+0000","last_undegraded":"2024-03-24T11:19:58.997443+0000","last_fullsized":"2024-03-24T11:19:58.997443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.033064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997683+0000","last_change":"2024-03-24T11:18:53.672450+0000","last_active":"2024-03-24T11:19:58.997683+0000","last_peered":"2024-03-24T11:19:58.997683+0000","last_clean":"2024-03-24T11:19:58.997683+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:19:58.997683+0000","last_undegraded":"2024-03-24T11:19:58.997683+0000","last_fullsized":"2024-03-24T11:19:58.997683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.274022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996737+0000","last_change":"2024-03-24T11:18:55.684714+0000","last_active":"2024-03-24T11:19:58.996737+0000","last_peered":"2024-03-24T11:19:58.996737+0000","last_clean":"2024-03-24T11:19:58.996737+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:19:58.996737+0000","last_undegraded":"2024-03-24T11:19:58.996737+0000","last_fullsized":"2024-03-24T11:19:58.996737+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.614052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470883+0000","last_change":"2024-03-24T11:18:59.707853+0000","last_active":"2024-03-24T11:19:59.470883+0000","last_peered":"2024-03-24T11:19:59.470883+0000","last_clean":"2024-03-24T11:19:59.470883+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:19:59.470883+0000","last_undegraded":"2024-03-24T11:19:59.470883+0000","last_fullsized":"2024-03-24T11:19:59.470883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.256196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:11.338029+0000","last_change":"2024-03-24T11:20:11.338061+0000","last_active":"2024-03-24T11:20:11.338029+0000","last_peered":"2024-03-24T11:20:11.338029+0000","last_clean":"2024-03-24T11:20:11.338029+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:20:11.338029+0000","last_undegraded":"2024-03-24T11:20:11.338029+0000","last_fullsized":"2024-03-24T11:20:11.338029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:20:11.337962+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:20:11.337962+0000","last_clean_scrub_stamp":"2024-03-24T11:20:11.337962+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:34.099110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:01.218166+0000","last_change":"2024-03-24T11:20:01.218213+0000","last_active":"2024-03-24T11:20:01.218166+0000","last_peered":"2024-03-24T11:20:01.218166+0000","last_clean":"2024-03-24T11:20:01.218166+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:20:01.218166+0000","last_undegraded":"2024-03-24T11:20:01.218166+0000","last_fullsized":"2024-03-24T11:20:01.218166+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:01.218061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.218061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.318777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.001142+0000","last_change":"2024-03-24T11:20:03.001173+0000","last_active":"2024-03-24T11:20:03.001142+0000","last_peered":"2024-03-24T11:20:03.001142+0000","last_clean":"2024-03-24T11:20:03.001142+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:20:03.001142+0000","last_undegraded":"2024-03-24T11:20:03.001142+0000","last_fullsized":"2024-03-24T11:20:03.001142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:20:03.001079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.001079+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.140666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.208905+0000","last_change":"2024-03-24T11:20:05.208928+0000","last_active":"2024-03-24T11:20:05.208905+0000","last_peered":"2024-03-24T11:20:05.208905+0000","last_clean":"2024-03-24T11:20:05.208905+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:20:05.208905+0000","last_undegraded":"2024-03-24T11:20:05.208905+0000","last_fullsized":"2024-03-24T11:20:05.208905+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.208832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.208832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:28.229456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:10.934318+0000","last_change":"2024-03-24T11:20:10.934360+0000","last_active":"2024-03-24T11:20:10.934318+0000","last_peered":"2024-03-24T11:20:10.934318+0000","last_clean":"2024-03-24T11:20:10.934318+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:20:10.934318+0000","last_undegraded":"2024-03-24T11:20:10.934318+0000","last_fullsized":"2024-03-24T11:20:10.934318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:10.934216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:10.934216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:33.120547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996459+0000","last_change":"2024-03-24T11:19:57.458406+0000","last_active":"2024-03-24T11:19:58.996459+0000","last_peered":"2024-03-24T11:19:58.996459+0000","last_clean":"2024-03-24T11:19:58.996459+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:19:58.996459+0000","last_undegraded":"2024-03-24T11:19:58.996459+0000","last_fullsized":"2024-03-24T11:19:58.996459+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:57.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:57.458291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.184542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:12.347796+0000","last_change":"2024-03-24T11:20:12.347839+0000","last_active":"2024-03-24T11:20:12.347796+0000","last_peered":"2024-03-24T11:20:12.347796+0000","last_clean":"2024-03-24T11:20:12.347796+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:20:12.347796+0000","last_undegraded":"2024-03-24T11:20:12.347796+0000","last_fullsized":"2024-03-24T11:20:12.347796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:20:12.347712+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:20:12.347712+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.347712+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.610126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.896322+0000","last_change":"2024-03-24T11:20:03.896346+0000","last_active":"2024-03-24T11:20:03.896322+0000","last_peered":"2024-03-24T11:20:03.896322+0000","last_clean":"2024-03-24T11:20:03.896322+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:20:03.896322+0000","last_undegraded":"2024-03-24T11:20:03.896322+0000","last_fullsized":"2024-03-24T11:20:03.896322+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.896225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.896225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.959693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470475+0000","last_change":"2024-03-24T11:18:57.700812+0000","last_active":"2024-03-24T11:19:59.470475+0000","last_peered":"2024-03-24T11:19:59.470475+0000","last_clean":"2024-03-24T11:19:59.470475+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:19:59.470475+0000","last_undegraded":"2024-03-24T11:19:59.470475+0000","last_fullsized":"2024-03-24T11:19:59.470475+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:25.753316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.018180+0000","last_change":"2024-03-24T11:20:07.018200+0000","last_active":"2024-03-24T11:20:07.018180+0000","last_peered":"2024-03-24T11:20:07.018180+0000","last_clean":"2024-03-24T11:20:07.018180+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:20:07.018180+0000","last_undegraded":"2024-03-24T11:20:07.018180+0000","last_fullsized":"2024-03-24T11:20:07.018180+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.018119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.018119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.136141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996480+0000","last_change":"2024-03-24T11:18:55.682010+0000","last_active":"2024-03-24T11:19:58.996480+0000","last_peered":"2024-03-24T11:19:58.996480+0000","last_clean":"2024-03-24T11:19:58.996480+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:19:58.996480+0000","last_undegraded":"2024-03-24T11:19:58.996480+0000","last_fullsized":"2024-03-24T11:19:58.996480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:03.491501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996465+0000","last_change":"2024-03-24T11:19:00.314378+0000","last_active":"2024-03-24T11:19:58.996465+0000","last_peered":"2024-03-24T11:19:58.996465+0000","last_clean":"2024-03-24T11:19:58.996465+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:19:58.996465+0000","last_undegraded":"2024-03-24T11:19:58.996465+0000","last_fullsized":"2024-03-24T11:19:58.996465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.875648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471763+0000","last_change":"2024-03-24T11:18:57.698486+0000","last_active":"2024-03-24T11:19:59.471763+0000","last_peered":"2024-03-24T11:19:59.471763+0000","last_clean":"2024-03-24T11:19:59.471763+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:19:59.471763+0000","last_undegraded":"2024-03-24T11:19:59.471763+0000","last_fullsized":"2024-03-24T11:19:59.471763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.798498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472405+0000","last_change":"2024-03-24T11:18:53.684878+0000","last_active":"2024-03-24T11:19:59.472405+0000","last_peered":"2024-03-24T11:19:59.472405+0000","last_clean":"2024-03-24T11:19:59.472405+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:19:59.472405+0000","last_undegraded":"2024-03-24T11:19:59.472405+0000","last_fullsized":"2024-03-24T11:19:59.472405+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.381938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996821+0000","last_change":"2024-03-24T11:18:56.141810+0000","last_active":"2024-03-24T11:19:58.996821+0000","last_peered":"2024-03-24T11:19:58.996821+0000","last_clean":"2024-03-24T11:19:58.996821+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:19:58.996821+0000","last_undegraded":"2024-03-24T11:19:58.996821+0000","last_fullsized":"2024-03-24T11:19:58.996821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.967564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.469986+0000","last_change":"2024-03-24T11:18:59.714114+0000","last_active":"2024-03-24T11:19:59.469986+0000","last_peered":"2024-03-24T11:19:59.469986+0000","last_clean":"2024-03-24T11:19:59.469986+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:19:59.469986+0000","last_undegraded":"2024-03-24T11:19:59.469986+0000","last_fullsized":"2024-03-24T11:19:59.469986+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.880814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996952+0000","last_change":"2024-03-24T11:18:57.695479+0000","last_active":"2024-03-24T11:19:58.996952+0000","last_peered":"2024-03-24T11:19:58.996952+0000","last_clean":"2024-03-24T11:19:58.996952+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:19:58.996952+0000","last_undegraded":"2024-03-24T11:19:58.996952+0000","last_fullsized":"2024-03-24T11:19:58.996952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.930508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471259+0000","last_change":"2024-03-24T11:18:53.681700+0000","last_active":"2024-03-24T11:19:59.471259+0000","last_peered":"2024-03-24T11:19:59.471259+0000","last_clean":"2024-03-24T11:19:59.471259+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:19:59.471259+0000","last_undegraded":"2024-03-24T11:19:59.471259+0000","last_fullsized":"2024-03-24T11:19:59.471259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.896550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471099+0000","last_change":"2024-03-24T11:18:55.683808+0000","last_active":"2024-03-24T11:19:59.471099+0000","last_peered":"2024-03-24T11:19:59.471099+0000","last_clean":"2024-03-24T11:19:59.471099+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:19:59.471099+0000","last_undegraded":"2024-03-24T11:19:59.471099+0000","last_fullsized":"2024-03-24T11:19:59.471099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.543772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.441848+0000","last_change":"2024-03-24T11:20:03.441868+0000","last_active":"2024-03-24T11:20:03.441848+0000","last_peered":"2024-03-24T11:20:03.441848+0000","last_clean":"2024-03-24T11:20:03.441848+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:20:03.441848+0000","last_undegraded":"2024-03-24T11:20:03.441848+0000","last_fullsized":"2024-03-24T11:20:03.441848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.441791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.441791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.538293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471008+0000","last_change":"2024-03-24T11:18:57.703012+0000","last_active":"2024-03-24T11:19:59.471008+0000","last_peered":"2024-03-24T11:19:59.471008+0000","last_clean":"2024-03-24T11:19:59.471008+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:19:59.471008+0000","last_undegraded":"2024-03-24T11:19:59.471008+0000","last_fullsized":"2024-03-24T11:19:59.471008+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.320906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997581+0000","last_change":"2024-03-24T11:19:56.100757+0000","last_active":"2024-03-24T11:19:58.997581+0000","last_peered":"2024-03-24T11:19:58.997581+0000","last_clean":"2024-03-24T11:19:58.997581+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:19:58.997581+0000","last_undegraded":"2024-03-24T11:19:58.997581+0000","last_fullsized":"2024-03-24T11:19:58.997581+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.100633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.072386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:04.036605+0000","last_change":"2024-03-24T11:20:04.036645+0000","last_active":"2024-03-24T11:20:04.036605+0000","last_peered":"2024-03-24T11:20:04.036605+0000","last_clean":"2024-03-24T11:20:04.036605+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:20:04.036605+0000","last_undegraded":"2024-03-24T11:20:04.036605+0000","last_fullsized":"2024-03-24T11:20:04.036605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:20:04.036510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.036510+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.183695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470693+0000","last_change":"2024-03-24T11:18:59.717539+0000","last_active":"2024-03-24T11:19:59.470693+0000","last_peered":"2024-03-24T11:19:59.470693+0000","last_clean":"2024-03-24T11:19:59.470693+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:19:59.470693+0000","last_undegraded":"2024-03-24T11:19:59.470693+0000","last_fullsized":"2024-03-24T11:19:59.470693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.956051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.390710+0000","last_change":"2024-03-24T11:20:07.390710+0000","last_active":"2024-03-24T11:20:07.390710+0000","last_peered":"2024-03-24T11:20:07.390710+0000","last_clean":"2024-03-24T11:20:07.390710+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:20:07.390710+0000","last_undegraded":"2024-03-24T11:20:07.390710+0000","last_fullsized":"2024-03-24T11:20:07.390710+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:07.390604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.390604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.892391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470985+0000","last_change":"2024-03-24T11:18:53.682165+0000","last_active":"2024-03-24T11:19:59.470985+0000","last_peered":"2024-03-24T11:19:59.470985+0000","last_clean":"2024-03-24T11:19:59.470985+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:19:59.470985+0000","last_undegraded":"2024-03-24T11:19:59.470985+0000","last_fullsized":"2024-03-24T11:19:59.470985+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.317744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997522+0000","last_change":"2024-03-24T11:18:56.141308+0000","last_active":"2024-03-24T11:19:58.997522+0000","last_peered":"2024-03-24T11:19:58.997522+0000","last_clean":"2024-03-24T11:19:58.997522+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:19:58.997522+0000","last_undegraded":"2024-03-24T11:19:58.997522+0000","last_fullsized":"2024-03-24T11:19:58.997522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.469993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998039+0000","last_change":"2024-03-24T11:18:59.705914+0000","last_active":"2024-03-24T11:19:58.998039+0000","last_peered":"2024-03-24T11:19:58.998039+0000","last_clean":"2024-03-24T11:19:58.998039+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:19:58.998039+0000","last_undegraded":"2024-03-24T11:19:58.998039+0000","last_fullsized":"2024-03-24T11:19:58.998039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.058567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:04.436584+0000","last_change":"2024-03-24T11:20:04.436584+0000","last_active":"2024-03-24T11:20:04.436584+0000","last_peered":"2024-03-24T11:20:04.436584+0000","last_clean":"2024-03-24T11:20:04.436584+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:20:04.436584+0000","last_undegraded":"2024-03-24T11:20:04.436584+0000","last_fullsized":"2024-03-24T11:20:04.436584+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:04.436493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.436493+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.892034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470274+0000","last_change":"2024-03-24T11:18:53.683017+0000","last_active":"2024-03-24T11:19:59.470274+0000","last_peered":"2024-03-24T11:19:59.470274+0000","last_clean":"2024-03-24T11:19:59.470274+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:19:59.470274+0000","last_undegraded":"2024-03-24T11:19:59.470274+0000","last_fullsized":"2024-03-24T11:19:59.470274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:14.179403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472787+0000","last_change":"2024-03-24T11:18:56.143374+0000","last_active":"2024-03-24T11:19:59.472787+0000","last_peered":"2024-03-24T11:19:59.472787+0000","last_clean":"2024-03-24T11:19:59.472787+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:19:59.472787+0000","last_undegraded":"2024-03-24T11:19:59.472787+0000","last_fullsized":"2024-03-24T11:19:59.472787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.504039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998349+0000","last_change":"2024-03-24T11:18:59.703086+0000","last_active":"2024-03-24T11:19:58.998349+0000","last_peered":"2024-03-24T11:19:58.998349+0000","last_clean":"2024-03-24T11:19:58.998349+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:19:58.998349+0000","last_undegraded":"2024-03-24T11:19:58.998349+0000","last_fullsized":"2024-03-24T11:19:58.998349+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.043850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997187+0000","last_change":"2024-03-24T11:18:57.694184+0000","last_active":"2024-03-24T11:19:58.997187+0000","last_peered":"2024-03-24T11:19:58.997187+0000","last_clean":"2024-03-24T11:19:58.997187+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:19:58.997187+0000","last_undegraded":"2024-03-24T11:19:58.997187+0000","last_fullsized":"2024-03-24T11:19:58.997187+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.074119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472417+0000","last_change":"2024-03-24T11:19:58.799434+0000","last_active":"2024-03-24T11:19:59.472417+0000","last_peered":"2024-03-24T11:19:59.472417+0000","last_clean":"2024-03-24T11:19:59.472417+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:19:59.472417+0000","last_undegraded":"2024-03-24T11:19:59.472417+0000","last_fullsized":"2024-03-24T11:19:59.472417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.799295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.799295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.674050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471090+0000","last_change":"2024-03-24T11:19:58.001386+0000","last_active":"2024-03-24T11:19:59.471090+0000","last_peered":"2024-03-24T11:19:59.471090+0000","last_clean":"2024-03-24T11:19:59.471090+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:19:59.471090+0000","last_undegraded":"2024-03-24T11:19:59.471090+0000","last_fullsized":"2024-03-24T11:19:59.471090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.001296+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.932223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471165+0000","last_change":"2024-03-24T11:19:00.314565+0000","last_active":"2024-03-24T11:19:59.471165+0000","last_peered":"2024-03-24T11:19:59.471165+0000","last_clean":"2024-03-24T11:19:59.471165+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:19:59.471165+0000","last_undegraded":"2024-03-24T11:19:59.471165+0000","last_fullsized":"2024-03-24T11:19:59.471165+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.091963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472513+0000","last_change":"2024-03-24T11:18:57.693993+0000","last_active":"2024-03-24T11:19:59.472513+0000","last_peered":"2024-03-24T11:19:59.472513+0000","last_clean":"2024-03-24T11:19:59.472513+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:19:59.472513+0000","last_undegraded":"2024-03-24T11:19:59.472513+0000","last_fullsized":"2024-03-24T11:19:59.472513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.747158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:01.231673+0000","last_change":"2024-03-24T11:20:01.231696+0000","last_active":"2024-03-24T11:20:01.231673+0000","last_peered":"2024-03-24T11:20:01.231673+0000","last_clean":"2024-03-24T11:20:01.231673+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:20:01.231673+0000","last_undegraded":"2024-03-24T11:20:01.231673+0000","last_fullsized":"2024-03-24T11:20:01.231673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:01.231617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.231617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.010965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997261+0000","last_change":"2024-03-24T11:18:56.142569+0000","last_active":"2024-03-24T11:19:58.997261+0000","last_peered":"2024-03-24T11:19:58.997261+0000","last_clean":"2024-03-24T11:19:58.997261+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:19:58.997261+0000","last_undegraded":"2024-03-24T11:19:58.997261+0000","last_fullsized":"2024-03-24T11:19:58.997261+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.660826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:02.408198+0000","last_change":"2024-03-24T11:20:02.408219+0000","last_active":"2024-03-24T11:20:02.408198+0000","last_peered":"2024-03-24T11:20:02.408198+0000","last_clean":"2024-03-24T11:20:02.408198+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:20:02.408198+0000","last_undegraded":"2024-03-24T11:20:02.408198+0000","last_fullsized":"2024-03-24T11:20:02.408198+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.408130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.408130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:03.752282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471342+0000","last_change":"2024-03-24T11:18:53.684612+0000","last_active":"2024-03-24T11:19:59.471342+0000","last_peered":"2024-03-24T11:19:59.471342+0000","last_clean":"2024-03-24T11:19:59.471342+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:19:59.471342+0000","last_undegraded":"2024-03-24T11:19:59.471342+0000","last_fullsized":"2024-03-24T11:19:59.471342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.251916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:12.354625+0000","last_change":"2024-03-24T11:20:12.354666+0000","last_active":"2024-03-24T11:20:12.354625+0000","last_peered":"2024-03-24T11:20:12.354625+0000","last_clean":"2024-03-24T11:20:12.354625+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:20:12.354625+0000","last_undegraded":"2024-03-24T11:20:12.354625+0000","last_fullsized":"2024-03-24T11:20:12.354625+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:20:12.354559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.354559+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.930978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.863734+0000","last_change":"2024-03-24T11:20:05.863734+0000","last_active":"2024-03-24T11:20:05.863734+0000","last_peered":"2024-03-24T11:20:05.863734+0000","last_clean":"2024-03-24T11:20:05.863734+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:20:05.863734+0000","last_undegraded":"2024-03-24T11:20:05.863734+0000","last_fullsized":"2024-03-24T11:20:05.863734+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:05.863616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.863616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:09.078050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996565+0000","last_change":"2024-03-24T11:18:59.706989+0000","last_active":"2024-03-24T11:19:58.996565+0000","last_peered":"2024-03-24T11:19:58.996565+0000","last_clean":"2024-03-24T11:19:58.996565+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:19:58.996565+0000","last_undegraded":"2024-03-24T11:19:58.996565+0000","last_fullsized":"2024-03-24T11:19:58.996565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.879815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472058+0000","last_change":"2024-03-24T11:18:53.682847+0000","last_active":"2024-03-24T11:19:59.472058+0000","last_peered":"2024-03-24T11:19:59.472058+0000","last_clean":"2024-03-24T11:19:59.472058+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:19:59.472058+0000","last_undegraded":"2024-03-24T11:19:59.472058+0000","last_fullsized":"2024-03-24T11:19:59.472058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.562909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997500+0000","last_change":"2024-03-24T11:18:55.683134+0000","last_active":"2024-03-24T11:19:58.997500+0000","last_peered":"2024-03-24T11:19:58.997500+0000","last_clean":"2024-03-24T11:19:58.997500+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:19:58.997500+0000","last_undegraded":"2024-03-24T11:19:58.997500+0000","last_fullsized":"2024-03-24T11:19:58.997500+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.994319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997439+0000","last_change":"2024-03-24T11:18:57.704315+0000","last_active":"2024-03-24T11:19:58.997439+0000","last_peered":"2024-03-24T11:19:58.997439+0000","last_clean":"2024-03-24T11:19:58.997439+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:19:58.997439+0000","last_undegraded":"2024-03-24T11:19:58.997439+0000","last_fullsized":"2024-03-24T11:19:58.997439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.986639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471181+0000","last_change":"2024-03-24T11:18:59.707271+0000","last_active":"2024-03-24T11:19:59.471181+0000","last_peered":"2024-03-24T11:19:59.471181+0000","last_clean":"2024-03-24T11:19:59.471181+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:19:59.471181+0000","last_undegraded":"2024-03-24T11:19:59.471181+0000","last_fullsized":"2024-03-24T11:19:59.471181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.298674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.998236+0000","last_change":"2024-03-24T11:18:57.699926+0000","last_active":"2024-03-24T11:19:58.998236+0000","last_peered":"2024-03-24T11:19:58.998236+0000","last_clean":"2024-03-24T11:19:58.998236+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:19:58.998236+0000","last_undegraded":"2024-03-24T11:19:58.998236+0000","last_fullsized":"2024-03-24T11:19:58.998236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:09.922131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.851778+0000","last_change":"2024-03-24T11:20:00.851820+0000","last_active":"2024-03-24T11:20:00.851778+0000","last_peered":"2024-03-24T11:20:00.851778+0000","last_clean":"2024-03-24T11:20:00.851778+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:20:00.851778+0000","last_undegraded":"2024-03-24T11:20:00.851778+0000","last_fullsized":"2024-03-24T11:20:00.851778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.851691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.851691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.342041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471767+0000","last_change":"2024-03-24T11:18:56.143415+0000","last_active":"2024-03-24T11:19:59.471767+0000","last_peered":"2024-03-24T11:19:59.471767+0000","last_clean":"2024-03-24T11:19:59.471767+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:19:59.471767+0000","last_undegraded":"2024-03-24T11:19:59.471767+0000","last_fullsized":"2024-03-24T11:19:59.471767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.105644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.980171+0000","last_change":"2024-03-24T11:20:07.980193+0000","last_active":"2024-03-24T11:20:07.980171+0000","last_peered":"2024-03-24T11:20:07.980171+0000","last_clean":"2024-03-24T11:20:07.980171+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:20:07.980171+0000","last_undegraded":"2024-03-24T11:20:07.980171+0000","last_fullsized":"2024-03-24T11:20:07.980171+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.980113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.980113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.058894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997984+0000","last_change":"2024-03-24T11:18:57.700062+0000","last_active":"2024-03-24T11:19:58.997984+0000","last_peered":"2024-03-24T11:19:58.997984+0000","last_clean":"2024-03-24T11:19:58.997984+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:19:58.997984+0000","last_undegraded":"2024-03-24T11:19:58.997984+0000","last_fullsized":"2024-03-24T11:19:58.997984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.041178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471983+0000","last_change":"2024-03-24T11:19:56.404250+0000","last_active":"2024-03-24T11:19:59.471983+0000","last_peered":"2024-03-24T11:19:59.471983+0000","last_clean":"2024-03-24T11:19:59.471983+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:19:59.471983+0000","last_undegraded":"2024-03-24T11:19:59.471983+0000","last_fullsized":"2024-03-24T11:19:59.471983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:19:56.404146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.404146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.193173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470638+0000","last_change":"2024-03-24T11:18:55.684166+0000","last_active":"2024-03-24T11:19:59.470638+0000","last_peered":"2024-03-24T11:19:59.470638+0000","last_clean":"2024-03-24T11:19:59.470638+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:19:59.470638+0000","last_undegraded":"2024-03-24T11:19:59.470638+0000","last_fullsized":"2024-03-24T11:19:59.470638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:03.187965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997025+0000","last_change":"2024-03-24T11:18:59.705914+0000","last_active":"2024-03-24T11:19:58.997025+0000","last_peered":"2024-03-24T11:19:58.997025+0000","last_clean":"2024-03-24T11:19:58.997025+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:19:58.997025+0000","last_undegraded":"2024-03-24T11:19:58.997025+0000","last_fullsized":"2024-03-24T11:19:58.997025+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:03.823429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471272+0000","last_change":"2024-03-24T11:18:56.680518+0000","last_active":"2024-03-24T11:19:59.471272+0000","last_peered":"2024-03-24T11:19:59.471272+0000","last_clean":"2024-03-24T11:19:59.471272+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:19:59.471272+0000","last_undegraded":"2024-03-24T11:19:59.471272+0000","last_fullsized":"2024-03-24T11:19:59.471272+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:18:51.653668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:13.269752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997102+0000","last_change":"2024-03-24T11:18:53.680619+0000","last_active":"2024-03-24T11:19:58.997102+0000","last_peered":"2024-03-24T11:19:58.997102+0000","last_clean":"2024-03-24T11:19:58.997102+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:19:58.997102+0000","last_undegraded":"2024-03-24T11:19:58.997102+0000","last_fullsized":"2024-03-24T11:19:58.997102+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.921367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":121,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.736489+0000","last_change":"2024-03-24T11:19:55.086981+0000","last_active":"2024-03-24T11:20:03.736489+0000","last_peered":"2024-03-24T11:20:03.736489+0000","last_clean":"2024-03-24T11:20:03.736489+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:20:03.736489+0000","last_undegraded":"2024-03-24T11:20:03.736489+0000","last_fullsized":"2024-03-24T11:20:03.736489+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:19:55.086847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:19:55.086847+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:14.068671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471091+0000","last_change":"2024-03-24T11:18:57.694386+0000","last_active":"2024-03-24T11:19:59.471091+0000","last_peered":"2024-03-24T11:19:59.471091+0000","last_clean":"2024-03-24T11:19:59.471091+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:19:59.471091+0000","last_undegraded":"2024-03-24T11:19:59.471091+0000","last_fullsized":"2024-03-24T11:19:59.471091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.661605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:12.833600+0000","last_change":"2024-03-24T11:20:12.833645+0000","last_active":"2024-03-24T11:20:12.833600+0000","last_peered":"2024-03-24T11:20:12.833600+0000","last_clean":"2024-03-24T11:20:12.833600+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:20:12.833600+0000","last_undegraded":"2024-03-24T11:20:12.833600+0000","last_fullsized":"2024-03-24T11:20:12.833600+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:12.833509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.833509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.607503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470446+0000","last_change":"2024-03-24T11:19:58.185937+0000","last_active":"2024-03-24T11:19:59.470446+0000","last_peered":"2024-03-24T11:19:59.470446+0000","last_clean":"2024-03-24T11:19:59.470446+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:19:59.470446+0000","last_undegraded":"2024-03-24T11:19:59.470446+0000","last_fullsized":"2024-03-24T11:19:59.470446+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.185832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.185832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.274032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.555970+0000","last_change":"2024-03-24T11:20:00.555990+0000","last_active":"2024-03-24T11:20:00.555970+0000","last_peered":"2024-03-24T11:20:00.555970+0000","last_clean":"2024-03-24T11:20:00.555970+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:20:00.555970+0000","last_undegraded":"2024-03-24T11:20:00.555970+0000","last_fullsized":"2024-03-24T11:20:00.555970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.555913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.555913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.222390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:02.552300+0000","last_change":"2024-03-24T11:20:02.552338+0000","last_active":"2024-03-24T11:20:02.552300+0000","last_peered":"2024-03-24T11:20:02.552300+0000","last_clean":"2024-03-24T11:20:02.552300+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:20:02.552300+0000","last_undegraded":"2024-03-24T11:20:02.552300+0000","last_fullsized":"2024-03-24T11:20:02.552300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:20:02.552224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.552224+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.289260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:08.530799+0000","last_change":"2024-03-24T11:20:08.530821+0000","last_active":"2024-03-24T11:20:08.530799+0000","last_peered":"2024-03-24T11:20:08.530799+0000","last_clean":"2024-03-24T11:20:08.530799+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:20:08.530799+0000","last_undegraded":"2024-03-24T11:20:08.530799+0000","last_fullsized":"2024-03-24T11:20:08.530799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:08.530740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:08.530740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.295185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472117+0000","last_change":"2024-03-24T11:18:59.707141+0000","last_active":"2024-03-24T11:19:59.472117+0000","last_peered":"2024-03-24T11:19:59.472117+0000","last_clean":"2024-03-24T11:19:59.472117+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:19:59.472117+0000","last_undegraded":"2024-03-24T11:19:59.472117+0000","last_fullsized":"2024-03-24T11:19:59.472117+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.755888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":306,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471422+0000","last_change":"2024-03-24T11:19:18.724688+0000","last_active":"2024-03-24T11:19:59.471422+0000","last_peered":"2024-03-24T11:19:59.471422+0000","last_clean":"2024-03-24T11:19:59.471422+0000","last_became_active":"2024-03-24T11:18:44.123994+0000","last_became_peered":"2024-03-24T11:18:44.123994+0000","last_unstale":"2024-03-24T11:19:59.471422+0000","last_undegraded":"2024-03-24T11:19:59.471422+0000","last_fullsized":"2024-03-24T11:19:59.471422+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-24T11:19:18.724552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:19:18.724552+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:46.099214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.392249+0000","last_change":"2024-03-24T11:20:05.392270+0000","last_active":"2024-03-24T11:20:05.392249+0000","last_peered":"2024-03-24T11:20:05.392249+0000","last_clean":"2024-03-24T11:20:05.392249+0000","last_became_active":"2024-03-24T11:18:53.678099+0000","last_became_peered":"2024-03-24T11:18:53.678099+0000","last_unstale":"2024-03-24T11:20:05.392249+0000","last_undegraded":"2024-03-24T11:20:05.392249+0000","last_fullsized":"2024-03-24T11:20:05.392249+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.392195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.392195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.644580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":87,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.736690+0000","last_change":"2024-03-24T11:18:56.141576+0000","last_active":"2024-03-24T11:20:03.736690+0000","last_peered":"2024-03-24T11:20:03.736690+0000","last_clean":"2024-03-24T11:20:03.736690+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:20:03.736690+0000","last_undegraded":"2024-03-24T11:20:03.736690+0000","last_fullsized":"2024-03-24T11:20:03.736690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.491233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:10.834617+0000","last_change":"2024-03-24T11:20:10.834643+0000","last_active":"2024-03-24T11:20:10.834617+0000","last_peered":"2024-03-24T11:20:10.834617+0000","last_clean":"2024-03-24T11:20:10.834617+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:20:10.834617+0000","last_undegraded":"2024-03-24T11:20:10.834617+0000","last_fullsized":"2024-03-24T11:20:10.834617+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:10.834537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:10.834537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:23.214600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471263+0000","last_change":"2024-03-24T11:18:59.710726+0000","last_active":"2024-03-24T11:19:59.471263+0000","last_peered":"2024-03-24T11:19:59.471263+0000","last_clean":"2024-03-24T11:19:59.471263+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:19:59.471263+0000","last_undegraded":"2024-03-24T11:19:59.471263+0000","last_fullsized":"2024-03-24T11:19:59.471263+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.585199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470639+0000","last_change":"2024-03-24T11:18:53.681515+0000","last_active":"2024-03-24T11:19:59.470639+0000","last_peered":"2024-03-24T11:19:59.470639+0000","last_clean":"2024-03-24T11:19:59.470639+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:19:59.470639+0000","last_undegraded":"2024-03-24T11:19:59.470639+0000","last_fullsized":"2024-03-24T11:19:59.470639+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:01.763672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":146,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:11.601832+0000","last_change":"2024-03-24T11:20:06.880339+0000","last_active":"2024-03-24T11:20:11.601832+0000","last_peered":"2024-03-24T11:20:11.601832+0000","last_clean":"2024-03-24T11:20:11.601832+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:20:11.601832+0000","last_undegraded":"2024-03-24T11:20:11.601832+0000","last_fullsized":"2024-03-24T11:20:11.601832+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:20:06.880224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.880224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:32.826706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470771+0000","last_change":"2024-03-24T11:18:55.681576+0000","last_active":"2024-03-24T11:19:59.470771+0000","last_peered":"2024-03-24T11:19:59.470771+0000","last_clean":"2024-03-24T11:19:59.470771+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:19:59.470771+0000","last_undegraded":"2024-03-24T11:19:59.470771+0000","last_fullsized":"2024-03-24T11:19:59.470771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.955414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472509+0000","last_change":"2024-03-24T11:18:57.693189+0000","last_active":"2024-03-24T11:19:59.472509+0000","last_peered":"2024-03-24T11:19:59.472509+0000","last_clean":"2024-03-24T11:19:59.472509+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:19:59.472509+0000","last_undegraded":"2024-03-24T11:19:59.472509+0000","last_fullsized":"2024-03-24T11:19:59.472509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.238110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470286+0000","last_change":"2024-03-24T11:19:00.312423+0000","last_active":"2024-03-24T11:19:59.470286+0000","last_peered":"2024-03-24T11:19:59.470286+0000","last_clean":"2024-03-24T11:19:59.470286+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:19:59.470286+0000","last_undegraded":"2024-03-24T11:19:59.470286+0000","last_fullsized":"2024-03-24T11:19:59.470286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:11.289224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:07.202646+0000","last_change":"2024-03-24T11:20:07.202667+0000","last_active":"2024-03-24T11:20:07.202646+0000","last_peered":"2024-03-24T11:20:07.202646+0000","last_clean":"2024-03-24T11:20:07.202646+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:20:07.202646+0000","last_undegraded":"2024-03-24T11:20:07.202646+0000","last_fullsized":"2024-03-24T11:20:07.202646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.202576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.202576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:10.625674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470502+0000","last_change":"2024-03-24T11:18:56.142025+0000","last_active":"2024-03-24T11:19:59.470502+0000","last_peered":"2024-03-24T11:19:59.470502+0000","last_clean":"2024-03-24T11:19:59.470502+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:19:59.470502+0000","last_undegraded":"2024-03-24T11:19:59.470502+0000","last_fullsized":"2024-03-24T11:19:59.470502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.374097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:05.186154+0000","last_change":"2024-03-24T11:20:05.186178+0000","last_active":"2024-03-24T11:20:05.186154+0000","last_peered":"2024-03-24T11:20:05.186154+0000","last_clean":"2024-03-24T11:20:05.186154+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:20:05.186154+0000","last_undegraded":"2024-03-24T11:20:05.186154+0000","last_fullsized":"2024-03-24T11:20:05.186154+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.186078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.695962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.469989+0000","last_change":"2024-03-24T11:18:59.714865+0000","last_active":"2024-03-24T11:19:59.469989+0000","last_peered":"2024-03-24T11:19:59.469989+0000","last_clean":"2024-03-24T11:19:59.469989+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:19:59.469989+0000","last_undegraded":"2024-03-24T11:19:59.469989+0000","last_fullsized":"2024-03-24T11:19:59.469989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:15.985683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:09.320150+0000","last_change":"2024-03-24T11:20:09.320186+0000","last_active":"2024-03-24T11:20:09.320150+0000","last_peered":"2024-03-24T11:20:09.320150+0000","last_clean":"2024-03-24T11:20:09.320150+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:20:09.320150+0000","last_undegraded":"2024-03-24T11:20:09.320150+0000","last_fullsized":"2024-03-24T11:20:09.320150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:09.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:09.320059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:11.542154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471059+0000","last_change":"2024-03-24T11:18:55.687126+0000","last_active":"2024-03-24T11:19:59.471059+0000","last_peered":"2024-03-24T11:19:59.471059+0000","last_clean":"2024-03-24T11:19:59.471059+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:19:59.471059+0000","last_undegraded":"2024-03-24T11:19:59.471059+0000","last_fullsized":"2024-03-24T11:19:59.471059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.723563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.736357+0000","last_change":"2024-03-24T11:20:01.039976+0000","last_active":"2024-03-24T11:20:03.736357+0000","last_peered":"2024-03-24T11:20:03.736357+0000","last_clean":"2024-03-24T11:20:03.736357+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:20:03.736357+0000","last_undegraded":"2024-03-24T11:20:03.736357+0000","last_fullsized":"2024-03-24T11:20:03.736357+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:01.039839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.039839+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:01.168501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471214+0000","last_change":"2024-03-24T11:19:00.314552+0000","last_active":"2024-03-24T11:19:59.471214+0000","last_peered":"2024-03-24T11:19:59.471214+0000","last_clean":"2024-03-24T11:19:59.471214+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:19:59.471214+0000","last_undegraded":"2024-03-24T11:19:59.471214+0000","last_fullsized":"2024-03-24T11:19:59.471214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.506599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":54,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.470940+0000","last_change":"2024-03-24T11:18:53.683466+0000","last_active":"2024-03-24T11:19:59.470940+0000","last_peered":"2024-03-24T11:19:59.470940+0000","last_clean":"2024-03-24T11:19:59.470940+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:19:59.470940+0000","last_undegraded":"2024-03-24T11:19:59.470940+0000","last_fullsized":"2024-03-24T11:19:59.470940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:10.603409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997139+0000","last_change":"2024-03-24T11:18:55.681171+0000","last_active":"2024-03-24T11:19:58.997139+0000","last_peered":"2024-03-24T11:19:58.997139+0000","last_clean":"2024-03-24T11:19:58.997139+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:19:58.997139+0000","last_undegraded":"2024-03-24T11:19:58.997139+0000","last_fullsized":"2024-03-24T11:19:58.997139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:05.814781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.736377+0000","last_change":"2024-03-24T11:18:57.697565+0000","last_active":"2024-03-24T11:20:03.736377+0000","last_peered":"2024-03-24T11:20:03.736377+0000","last_clean":"2024-03-24T11:20:03.736377+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:20:03.736377+0000","last_undegraded":"2024-03-24T11:20:03.736377+0000","last_fullsized":"2024-03-24T11:20:03.736377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.519845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997134+0000","last_change":"2024-03-24T11:18:59.706057+0000","last_active":"2024-03-24T11:19:58.997134+0000","last_peered":"2024-03-24T11:19:58.997134+0000","last_clean":"2024-03-24T11:19:58.997134+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:19:58.997134+0000","last_undegraded":"2024-03-24T11:19:58.997134+0000","last_fullsized":"2024-03-24T11:19:58.997134+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.016361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997004+0000","last_change":"2024-03-24T11:18:53.679203+0000","last_active":"2024-03-24T11:19:58.997004+0000","last_peered":"2024-03-24T11:19:58.997004+0000","last_clean":"2024-03-24T11:19:58.997004+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:19:58.997004+0000","last_undegraded":"2024-03-24T11:19:58.997004+0000","last_fullsized":"2024-03-24T11:19:58.997004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.063996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997088+0000","last_change":"2024-03-24T11:18:56.141573+0000","last_active":"2024-03-24T11:19:58.997088+0000","last_peered":"2024-03-24T11:19:58.997088+0000","last_clean":"2024-03-24T11:19:58.997088+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:19:58.997088+0000","last_undegraded":"2024-03-24T11:19:58.997088+0000","last_fullsized":"2024-03-24T11:19:58.997088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.850624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:00.029170+0000","last_change":"2024-03-24T11:20:00.029193+0000","last_active":"2024-03-24T11:20:00.029170+0000","last_peered":"2024-03-24T11:20:00.029170+0000","last_clean":"2024-03-24T11:20:00.029170+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:20:00.029170+0000","last_undegraded":"2024-03-24T11:20:00.029170+0000","last_fullsized":"2024-03-24T11:20:00.029170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.029091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.029091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.480294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472484+0000","last_change":"2024-03-24T11:18:59.715347+0000","last_active":"2024-03-24T11:19:59.472484+0000","last_peered":"2024-03-24T11:19:59.472484+0000","last_clean":"2024-03-24T11:19:59.472484+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:19:59.472484+0000","last_undegraded":"2024-03-24T11:19:59.472484+0000","last_fullsized":"2024-03-24T11:19:59.472484+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.582227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.736555+0000","last_change":"2024-03-24T11:18:57.699380+0000","last_active":"2024-03-24T11:20:03.736555+0000","last_peered":"2024-03-24T11:20:03.736555+0000","last_clean":"2024-03-24T11:20:03.736555+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:20:03.736555+0000","last_undegraded":"2024-03-24T11:20:03.736555+0000","last_fullsized":"2024-03-24T11:20:03.736555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.138955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:02.414959+0000","last_change":"2024-03-24T11:20:02.414997+0000","last_active":"2024-03-24T11:20:02.414959+0000","last_peered":"2024-03-24T11:20:02.414959+0000","last_clean":"2024-03-24T11:20:02.414959+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:20:02.414959+0000","last_undegraded":"2024-03-24T11:20:02.414959+0000","last_fullsized":"2024-03-24T11:20:02.414959+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.414845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.414845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.613098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.471783+0000","last_change":"2024-03-24T11:18:55.686410+0000","last_active":"2024-03-24T11:19:59.471783+0000","last_peered":"2024-03-24T11:19:59.471783+0000","last_clean":"2024-03-24T11:19:59.471783+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:19:59.471783+0000","last_undegraded":"2024-03-24T11:19:59.471783+0000","last_fullsized":"2024-03-24T11:19:59.471783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997319+0000","last_change":"2024-03-24T11:19:00.312496+0000","last_active":"2024-03-24T11:19:58.997319+0000","last_peered":"2024-03-24T11:19:58.997319+0000","last_clean":"2024-03-24T11:19:58.997319+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:19:58.997319+0000","last_undegraded":"2024-03-24T11:19:58.997319+0000","last_fullsized":"2024-03-24T11:19:58.997319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.512245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:59.472481+0000","last_change":"2024-03-24T11:18:57.696783+0000","last_active":"2024-03-24T11:19:59.472481+0000","last_peered":"2024-03-24T11:19:59.472481+0000","last_clean":"2024-03-24T11:19:59.472481+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:19:59.472481+0000","last_undegraded":"2024-03-24T11:19:59.472481+0000","last_fullsized":"2024-03-24T11:19:59.472481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.106375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.996975+0000","last_change":"2024-03-24T11:18:53.679465+0000","last_active":"2024-03-24T11:19:58.996975+0000","last_peered":"2024-03-24T11:19:58.996975+0000","last_clean":"2024-03-24T11:19:58.996975+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:19:58.996975+0000","last_undegraded":"2024-03-24T11:19:58.996975+0000","last_fullsized":"2024-03-24T11:19:58.996975+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.456813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997636+0000","last_change":"2024-03-24T11:18:55.684671+0000","last_active":"2024-03-24T11:19:58.997636+0000","last_peered":"2024-03-24T11:19:58.997636+0000","last_clean":"2024-03-24T11:19:58.997636+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:19:58.997636+0000","last_undegraded":"2024-03-24T11:19:58.997636+0000","last_fullsized":"2024-03-24T11:19:58.997636+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:08.694354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:06.548093+0000","last_change":"2024-03-24T11:20:06.548123+0000","last_active":"2024-03-24T11:20:06.548093+0000","last_peered":"2024-03-24T11:20:06.548093+0000","last_clean":"2024-03-24T11:20:06.548093+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:20:06.548093+0000","last_undegraded":"2024-03-24T11:20:06.548093+0000","last_fullsized":"2024-03-24T11:20:06.548093+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:06.548036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.548036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:08.537924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:04.159001+0000","last_change":"2024-03-24T11:20:04.159022+0000","last_active":"2024-03-24T11:20:04.159001+0000","last_peered":"2024-03-24T11:20:04.159001+0000","last_clean":"2024-03-24T11:20:04.159001+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:20:04.159001+0000","last_undegraded":"2024-03-24T11:20:04.159001+0000","last_fullsized":"2024-03-24T11:20:04.159001+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:04.158945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.158945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.212380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.233296+0000","last_change":"2024-03-24T11:20:03.233345+0000","last_active":"2024-03-24T11:20:03.233296+0000","last_peered":"2024-03-24T11:20:03.233296+0000","last_clean":"2024-03-24T11:20:03.233296+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:20:03.233296+0000","last_undegraded":"2024-03-24T11:20:03.233296+0000","last_fullsized":"2024-03-24T11:20:03.233296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:03.233200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.233200+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.734595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997689+0000","last_change":"2024-03-24T11:19:18.829942+0000","last_active":"2024-03-24T11:19:58.997689+0000","last_peered":"2024-03-24T11:19:58.997689+0000","last_clean":"2024-03-24T11:19:58.997689+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:19:58.997689+0000","last_undegraded":"2024-03-24T11:19:58.997689+0000","last_fullsized":"2024-03-24T11:19:58.997689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.426875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997890+0000","last_change":"2024-03-24T11:18:59.703044+0000","last_active":"2024-03-24T11:19:58.997890+0000","last_peered":"2024-03-24T11:19:58.997890+0000","last_clean":"2024-03-24T11:19:58.997890+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:19:58.997890+0000","last_undegraded":"2024-03-24T11:19:58.997890+0000","last_fullsized":"2024-03-24T11:19:58.997890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:07.281189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:20:03.736399+0000","last_change":"2024-03-24T11:18:57.701581+0000","last_active":"2024-03-24T11:20:03.736399+0000","last_peered":"2024-03-24T11:20:03.736399+0000","last_clean":"2024-03-24T11:20:03.736399+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:20:03.736399+0000","last_undegraded":"2024-03-24T11:20:03.736399+0000","last_fullsized":"2024-03-24T11:20:03.736399+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.273789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997367+0000","last_change":"2024-03-24T11:18:53.685545+0000","last_active":"2024-03-24T11:19:58.997367+0000","last_peered":"2024-03-24T11:19:58.997367+0000","last_clean":"2024-03-24T11:19:58.997367+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:19:58.997367+0000","last_undegraded":"2024-03-24T11:19:58.997367+0000","last_fullsized":"2024-03-24T11:19:58.997367+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:06.153480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T11:19:58.997299+0000","last_change":"2024-03-24T11:18:55.680886+0000","last_active":"2024-03-24T11:19:58.997299+0000","last_peered":"2024-03-24T11:19:58.997299+0000","last_clean":"2024-03-24T11:19:58.997299+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:19:58.997299+0000","last_undegraded":"2024-03-24T11:19:58.997299+0000","last_fullsized":"2024-03-24T11:19:58.997299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:16.995747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":402,"ondisk_log_size":402,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070484,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32624,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712528,"statfs":{"total":95995035648,"available":95961628672,"internally_reserved":0,"allocated":1482752,"data_stored":1004988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364825,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32612,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712540,"statfs":{"total":95995035648,"available":95961640960,"internally_reserved":0,"allocated":1470464,"data_stored":1005209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":415678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32632,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712520,"statfs":{"total":95995035648,"available":95961620480,"internally_reserved":0,"allocated":1490944,"data_stored":1004743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":414136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":414870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":901120,"data_stored":415048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659166,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":414402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:20:17.744 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:20:17.744 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:20:17.744 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:20:17.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:17 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:17.985 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:17 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:19.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:20:19.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:20:19.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:20:19.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:20:19.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:20:19.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:19.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:19.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:20:19.123 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:18 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:20:19.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:20:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:20:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:20:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:20:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T11:20:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:20:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:18 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:20:19.920 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:20:19.920 INFO:teuthology.orchestra.run.smithi179.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:20:20.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:19 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:20:20.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:19 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:20:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:19 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:20:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:19 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:20:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:20:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:20:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:20:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:20:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:20:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:20:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:20:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:19 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:20:20.618 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:20:20.619 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:20:20.619 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-24T11:20:20.631 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-24T11:20:20.631 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': 'f73f1dfd96849300c4322bd7259a3ab37f94c200'}} 2024-03-24T11:20:20.632 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-24T11:20:20.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-24T11:20:20.874 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:20:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:20:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3125706436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:20:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[29608]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:20:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:20:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:20:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:20:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3125706436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:20:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[37907]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:20:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:20 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:20:21.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:20 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:20:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:20 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:20:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:20 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3125706436' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:20:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:20 smithi181 ceph-mon[33998]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:20 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:20:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:20 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:20:22.031 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:20:22.032 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:20:22.033 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:20:22.033 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:21 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:20:22.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:21 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:20:22.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:21 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:20:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:21 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:20:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:21 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:20:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:21 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:20:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:21 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:20:23.052 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:22 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:20:23.052 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:22 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:20:23.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:22 smithi181 ceph-mon[33998]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-mon[29608]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-mon[37907]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:22] ENGINE Bus STOPPING 2024-03-24T11:20:23.121 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:20:23.122 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:22] ENGINE Bus STOPPED 2024-03-24T11:20:23.122 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:22] ENGINE Bus STARTING 2024-03-24T11:20:23.122 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:22] ENGINE Serving on http://:::9283 2024-03-24T11:20:23.122 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:20:22] ENGINE Bus STARTED 2024-03-24T11:20:23.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:20:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:20:23.567 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-24T11:20:23.567 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:20:23.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:20:23.814 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:23.937 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:20:23.937 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:20:23.937 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' 2024-03-24T11:20:23.937 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[29608]: 6.1c deep-scrub starts 2024-03-24T11:20:23.937 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[29608]: 6.1c deep-scrub ok 2024-03-24T11:20:23.937 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:23.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:20:23.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:20:23.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' 2024-03-24T11:20:23.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[37907]: 6.1c deep-scrub starts 2024-03-24T11:20:23.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[37907]: 6.1c deep-scrub ok 2024-03-24T11:20:23.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:23 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:24.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:23 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:20:24.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:23 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:20:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' 2024-03-24T11:20:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:23 smithi181 ceph-mon[33998]: 6.1c deep-scrub starts 2024-03-24T11:20:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:23 smithi181 ceph-mon[33998]: 6.1c deep-scrub ok 2024-03-24T11:20:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:23 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:24.937 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[72689]: ts=2024-03-24T11:20:24.587Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002396578s 2024-03-24T11:20:24.937 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:20:24.937 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:20:24.937 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 3.7 deep-scrub starts 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 3.7 deep-scrub ok 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 4.a deep-scrub starts 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 4.a deep-scrub ok 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 4.d deep-scrub starts 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[37907]: 4.d deep-scrub ok 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:20:24.938 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 3.7 deep-scrub starts 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 3.7 deep-scrub ok 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 4.a deep-scrub starts 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 4.a deep-scrub ok 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 4.d deep-scrub starts 2024-03-24T11:20:24.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:24 smithi179 ceph-mon[29608]: 4.d deep-scrub ok 2024-03-24T11:20:25.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:20:25.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 3.7 deep-scrub starts 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 3.7 deep-scrub ok 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 4.a deep-scrub starts 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 4.a deep-scrub ok 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 4.d deep-scrub starts 2024-03-24T11:20:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:24 smithi181 ceph-mon[33998]: 4.d deep-scrub ok 2024-03-24T11:20:25.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:20:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:20:25.661 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:20:25.663 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:20:25.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:25 smithi181 ceph-mon[33998]: 3.b scrub starts 2024-03-24T11:20:25.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:25 smithi181 ceph-mon[33998]: 3.b scrub ok 2024-03-24T11:20:25.916 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:25 smithi181 ceph-mon[33998]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:25.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:25 smithi179 ceph-mon[29608]: 3.b scrub starts 2024-03-24T11:20:25.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:25 smithi179 ceph-mon[29608]: 3.b scrub ok 2024-03-24T11:20:25.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:25 smithi179 ceph-mon[29608]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:25.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:25 smithi179 ceph-mon[37907]: 3.b scrub starts 2024-03-24T11:20:25.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:25 smithi179 ceph-mon[37907]: 3.b scrub ok 2024-03-24T11:20:25.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:25 smithi179 ceph-mon[37907]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:26.387 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-24T11:20:23.988956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1016,"num_read_kb":862,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":613,"ondisk_log_size":613,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249204,"kb_used_data":7868,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749712012,"statfs":{"total":767960285184,"available":767705100288,"internally_reserved":0,"allocated":8056832,"data_stored":4757546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002619"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717940+0000","last_change":"2024-03-24T11:18:59.704400+0000","last_active":"2024-03-24T11:20:17.717940+0000","last_peered":"2024-03-24T11:20:17.717940+0000","last_clean":"2024-03-24T11:20:17.717940+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:20:17.717940+0000","last_undegraded":"2024-03-24T11:20:17.717940+0000","last_fullsized":"2024-03-24T11:20:17.717940+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:22.357471+0000","last_change":"2024-03-24T11:20:22.357494+0000","last_active":"2024-03-24T11:20:22.357471+0000","last_peered":"2024-03-24T11:20:22.357471+0000","last_clean":"2024-03-24T11:20:22.357471+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:20:22.357471+0000","last_undegraded":"2024-03-24T11:20:22.357471+0000","last_fullsized":"2024-03-24T11:20:22.357471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:22.357393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:22.357393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:44.024523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721003+0000","last_change":"2024-03-24T11:18:53.684875+0000","last_active":"2024-03-24T11:20:17.721003+0000","last_peered":"2024-03-24T11:20:17.721003+0000","last_clean":"2024-03-24T11:20:17.721003+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:20:17.721003+0000","last_undegraded":"2024-03-24T11:20:17.721003+0000","last_fullsized":"2024-03-24T11:20:17.721003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:18.084479+0000","last_change":"2024-03-24T11:20:18.084538+0000","last_active":"2024-03-24T11:20:18.084479+0000","last_peered":"2024-03-24T11:20:18.084479+0000","last_clean":"2024-03-24T11:20:18.084479+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:20:18.084479+0000","last_undegraded":"2024-03-24T11:20:18.084479+0000","last_fullsized":"2024-03-24T11:20:18.084479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:20:18.084379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:18.084379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:40.748807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727934+0000","last_change":"2024-03-24T11:19:00.314493+0000","last_active":"2024-03-24T11:20:17.727934+0000","last_peered":"2024-03-24T11:20:17.727934+0000","last_clean":"2024-03-24T11:20:17.727934+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:20:17.727934+0000","last_undegraded":"2024-03-24T11:20:17.727934+0000","last_fullsized":"2024-03-24T11:20:17.727934+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.788301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717485+0000","last_change":"2024-03-24T11:18:57.698383+0000","last_active":"2024-03-24T11:20:17.717485+0000","last_peered":"2024-03-24T11:20:17.717485+0000","last_clean":"2024-03-24T11:20:17.717485+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:20:17.717485+0000","last_undegraded":"2024-03-24T11:20:17.717485+0000","last_fullsized":"2024-03-24T11:20:17.717485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728525+0000","last_change":"2024-03-24T11:18:53.680266+0000","last_active":"2024-03-24T11:20:17.728525+0000","last_peered":"2024-03-24T11:20:17.728525+0000","last_clean":"2024-03-24T11:20:17.728525+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:20:17.728525+0000","last_undegraded":"2024-03-24T11:20:17.728525+0000","last_fullsized":"2024-03-24T11:20:17.728525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727754+0000","last_change":"2024-03-24T11:18:55.680984+0000","last_active":"2024-03-24T11:20:17.727754+0000","last_peered":"2024-03-24T11:20:17.727754+0000","last_clean":"2024-03-24T11:20:17.727754+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:20:17.727754+0000","last_undegraded":"2024-03-24T11:20:17.727754+0000","last_fullsized":"2024-03-24T11:20:17.727754+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:24.173013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.068322+0000","last_change":"2024-03-24T11:20:20.068342+0000","last_active":"2024-03-24T11:20:20.068322+0000","last_peered":"2024-03-24T11:20:20.068322+0000","last_clean":"2024-03-24T11:20:20.068322+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:20:20.068322+0000","last_undegraded":"2024-03-24T11:20:20.068322+0000","last_fullsized":"2024-03-24T11:20:20.068322+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:20.068266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.068266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:35.574643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724680+0000","last_change":"2024-03-24T11:20:05.382085+0000","last_active":"2024-03-24T11:20:17.724680+0000","last_peered":"2024-03-24T11:20:17.724680+0000","last_clean":"2024-03-24T11:20:17.724680+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:20:17.724680+0000","last_undegraded":"2024-03-24T11:20:17.724680+0000","last_fullsized":"2024-03-24T11:20:17.724680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.381936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.381936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.568574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:18.374322+0000","last_change":"2024-03-24T11:20:18.374342+0000","last_active":"2024-03-24T11:20:18.374322+0000","last_peered":"2024-03-24T11:20:18.374322+0000","last_clean":"2024-03-24T11:20:18.374322+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:20:18.374322+0000","last_undegraded":"2024-03-24T11:20:18.374322+0000","last_fullsized":"2024-03-24T11:20:18.374322+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:18.374262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:18.374262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:43.607532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718059+0000","last_change":"2024-03-24T11:18:55.682437+0000","last_active":"2024-03-24T11:20:17.718059+0000","last_peered":"2024-03-24T11:20:17.718059+0000","last_clean":"2024-03-24T11:20:17.718059+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:20:17.718059+0000","last_undegraded":"2024-03-24T11:20:17.718059+0000","last_fullsized":"2024-03-24T11:20:17.718059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.808265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724501+0000","last_change":"2024-03-24T11:18:59.707100+0000","last_active":"2024-03-24T11:20:17.724501+0000","last_peered":"2024-03-24T11:20:17.724501+0000","last_clean":"2024-03-24T11:20:17.724501+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:20:17.724501+0000","last_undegraded":"2024-03-24T11:20:17.724501+0000","last_fullsized":"2024-03-24T11:20:17.724501+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725164+0000","last_change":"2024-03-24T11:18:57.701380+0000","last_active":"2024-03-24T11:20:17.725164+0000","last_peered":"2024-03-24T11:20:17.725164+0000","last_clean":"2024-03-24T11:20:17.725164+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:20:17.725164+0000","last_undegraded":"2024-03-24T11:20:17.725164+0000","last_fullsized":"2024-03-24T11:20:17.725164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724758+0000","last_change":"2024-03-24T11:18:53.681681+0000","last_active":"2024-03-24T11:20:17.724758+0000","last_peered":"2024-03-24T11:20:17.724758+0000","last_clean":"2024-03-24T11:20:17.724758+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:20:17.724758+0000","last_undegraded":"2024-03-24T11:20:17.724758+0000","last_fullsized":"2024-03-24T11:20:17.724758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718005+0000","last_change":"2024-03-24T11:18:55.686558+0000","last_active":"2024-03-24T11:20:17.718005+0000","last_peered":"2024-03-24T11:20:17.718005+0000","last_clean":"2024-03-24T11:20:17.718005+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:20:17.718005+0000","last_undegraded":"2024-03-24T11:20:17.718005+0000","last_fullsized":"2024-03-24T11:20:17.718005+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.670081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718719+0000","last_change":"2024-03-24T11:20:00.997176+0000","last_active":"2024-03-24T11:20:17.718719+0000","last_peered":"2024-03-24T11:20:17.718719+0000","last_clean":"2024-03-24T11:20:17.718719+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:20:17.718719+0000","last_undegraded":"2024-03-24T11:20:17.718719+0000","last_fullsized":"2024-03-24T11:20:17.718719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.997096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.997096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.143056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720937+0000","last_change":"2024-03-24T11:20:15.352309+0000","last_active":"2024-03-24T11:20:17.720937+0000","last_peered":"2024-03-24T11:20:17.720937+0000","last_clean":"2024-03-24T11:20:17.720937+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:20:17.720937+0000","last_undegraded":"2024-03-24T11:20:17.720937+0000","last_fullsized":"2024-03-24T11:20:17.720937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:15.352224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:15.352224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:36.853195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721464+0000","last_change":"2024-03-24T11:18:53.681845+0000","last_active":"2024-03-24T11:20:17.721464+0000","last_peered":"2024-03-24T11:20:17.721464+0000","last_clean":"2024-03-24T11:20:17.721464+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:20:17.721464+0000","last_undegraded":"2024-03-24T11:20:17.721464+0000","last_fullsized":"2024-03-24T11:20:17.721464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724932+0000","last_change":"2024-03-24T11:20:11.516126+0000","last_active":"2024-03-24T11:20:17.724932+0000","last_peered":"2024-03-24T11:20:17.724932+0000","last_clean":"2024-03-24T11:20:17.724932+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:20:17.724932+0000","last_undegraded":"2024-03-24T11:20:17.724932+0000","last_fullsized":"2024-03-24T11:20:17.724932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:20:11.516026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:11.516026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.641243+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728352+0000","last_change":"2024-03-24T11:19:00.312267+0000","last_active":"2024-03-24T11:20:17.728352+0000","last_peered":"2024-03-24T11:20:17.728352+0000","last_clean":"2024-03-24T11:20:17.728352+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:20:17.728352+0000","last_undegraded":"2024-03-24T11:20:17.728352+0000","last_fullsized":"2024-03-24T11:20:17.728352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.807373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718115+0000","last_change":"2024-03-24T11:18:57.699401+0000","last_active":"2024-03-24T11:20:17.718115+0000","last_peered":"2024-03-24T11:20:17.718115+0000","last_clean":"2024-03-24T11:20:17.718115+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:20:17.718115+0000","last_undegraded":"2024-03-24T11:20:17.718115+0000","last_fullsized":"2024-03-24T11:20:17.718115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718081+0000","last_change":"2024-03-24T11:18:53.672450+0000","last_active":"2024-03-24T11:20:17.718081+0000","last_peered":"2024-03-24T11:20:17.718081+0000","last_clean":"2024-03-24T11:20:17.718081+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:20:17.718081+0000","last_undegraded":"2024-03-24T11:20:17.718081+0000","last_fullsized":"2024-03-24T11:20:17.718081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728180+0000","last_change":"2024-03-24T11:18:55.684714+0000","last_active":"2024-03-24T11:20:17.728180+0000","last_peered":"2024-03-24T11:20:17.728180+0000","last_clean":"2024-03-24T11:20:17.728180+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:20:17.728180+0000","last_undegraded":"2024-03-24T11:20:17.728180+0000","last_fullsized":"2024-03-24T11:20:17.728180+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.614052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:22.871913+0000","last_change":"2024-03-24T11:20:22.871965+0000","last_active":"2024-03-24T11:20:22.871913+0000","last_peered":"2024-03-24T11:20:22.871913+0000","last_clean":"2024-03-24T11:20:22.871913+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:20:22.871913+0000","last_undegraded":"2024-03-24T11:20:22.871913+0000","last_fullsized":"2024-03-24T11:20:22.871913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:22.871822+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:45.749060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727563+0000","last_change":"2024-03-24T11:20:11.338061+0000","last_active":"2024-03-24T11:20:17.727563+0000","last_peered":"2024-03-24T11:20:17.727563+0000","last_clean":"2024-03-24T11:20:17.727563+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:20:17.727563+0000","last_undegraded":"2024-03-24T11:20:17.727563+0000","last_fullsized":"2024-03-24T11:20:17.727563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:20:11.337962+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:20:11.337962+0000","last_clean_scrub_stamp":"2024-03-24T11:20:11.337962+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:34.099110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722239+0000","last_change":"2024-03-24T11:20:01.218213+0000","last_active":"2024-03-24T11:20:17.722239+0000","last_peered":"2024-03-24T11:20:17.722239+0000","last_clean":"2024-03-24T11:20:17.722239+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:20:17.722239+0000","last_undegraded":"2024-03-24T11:20:17.722239+0000","last_fullsized":"2024-03-24T11:20:17.722239+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:01.218061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.218061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.318777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721941+0000","last_change":"2024-03-24T11:20:03.001173+0000","last_active":"2024-03-24T11:20:17.721941+0000","last_peered":"2024-03-24T11:20:17.721941+0000","last_clean":"2024-03-24T11:20:17.721941+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:20:17.721941+0000","last_undegraded":"2024-03-24T11:20:17.721941+0000","last_fullsized":"2024-03-24T11:20:17.721941+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:20:03.001079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.001079+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.140666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721084+0000","last_change":"2024-03-24T11:20:05.208928+0000","last_active":"2024-03-24T11:20:17.721084+0000","last_peered":"2024-03-24T11:20:17.721084+0000","last_clean":"2024-03-24T11:20:17.721084+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:20:17.721084+0000","last_undegraded":"2024-03-24T11:20:17.721084+0000","last_fullsized":"2024-03-24T11:20:17.721084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.208832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.208832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:28.229456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722066+0000","last_change":"2024-03-24T11:20:10.934360+0000","last_active":"2024-03-24T11:20:17.722066+0000","last_peered":"2024-03-24T11:20:17.722066+0000","last_clean":"2024-03-24T11:20:17.722066+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:20:17.722066+0000","last_undegraded":"2024-03-24T11:20:17.722066+0000","last_fullsized":"2024-03-24T11:20:17.722066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:10.934216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:10.934216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:33.120547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727493+0000","last_change":"2024-03-24T11:19:57.458406+0000","last_active":"2024-03-24T11:20:17.727493+0000","last_peered":"2024-03-24T11:20:17.727493+0000","last_clean":"2024-03-24T11:20:17.727493+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:20:17.727493+0000","last_undegraded":"2024-03-24T11:20:17.727493+0000","last_fullsized":"2024-03-24T11:20:17.727493+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:57.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:57.458291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.184542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":83,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727447+0000","last_change":"2024-03-24T11:20:12.347839+0000","last_active":"2024-03-24T11:20:17.727447+0000","last_peered":"2024-03-24T11:20:17.727447+0000","last_clean":"2024-03-24T11:20:17.727447+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:20:17.727447+0000","last_undegraded":"2024-03-24T11:20:17.727447+0000","last_fullsized":"2024-03-24T11:20:17.727447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:20:12.347712+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:20:12.347712+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.347712+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.610126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725064+0000","last_change":"2024-03-24T11:20:03.896346+0000","last_active":"2024-03-24T11:20:17.725064+0000","last_peered":"2024-03-24T11:20:17.725064+0000","last_clean":"2024-03-24T11:20:17.725064+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:20:17.725064+0000","last_undegraded":"2024-03-24T11:20:17.725064+0000","last_fullsized":"2024-03-24T11:20:17.725064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.896225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.896225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.959693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725295+0000","last_change":"2024-03-24T11:18:57.700812+0000","last_active":"2024-03-24T11:20:17.725295+0000","last_peered":"2024-03-24T11:20:17.725295+0000","last_clean":"2024-03-24T11:20:17.725295+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:20:17.725295+0000","last_undegraded":"2024-03-24T11:20:17.725295+0000","last_fullsized":"2024-03-24T11:20:17.725295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:25.753316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720979+0000","last_change":"2024-03-24T11:20:07.018200+0000","last_active":"2024-03-24T11:20:17.720979+0000","last_peered":"2024-03-24T11:20:17.720979+0000","last_clean":"2024-03-24T11:20:17.720979+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:20:17.720979+0000","last_undegraded":"2024-03-24T11:20:17.720979+0000","last_fullsized":"2024-03-24T11:20:17.720979+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.018119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.018119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.136141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"87'6","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727499+0000","last_change":"2024-03-24T11:20:14.327248+0000","last_active":"2024-03-24T11:20:17.727499+0000","last_peered":"2024-03-24T11:20:17.727499+0000","last_clean":"2024-03-24T11:20:17.727499+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:20:17.727499+0000","last_undegraded":"2024-03-24T11:20:17.727499+0000","last_fullsized":"2024-03-24T11:20:17.727499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'6","last_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:20:14.327154+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:35.752212+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727829+0000","last_change":"2024-03-24T11:19:00.314378+0000","last_active":"2024-03-24T11:20:17.727829+0000","last_peered":"2024-03-24T11:20:17.727829+0000","last_clean":"2024-03-24T11:20:17.727829+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:20:17.727829+0000","last_undegraded":"2024-03-24T11:20:17.727829+0000","last_fullsized":"2024-03-24T11:20:17.727829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727165+0000","last_change":"2024-03-24T11:18:57.698486+0000","last_active":"2024-03-24T11:20:17.727165+0000","last_peered":"2024-03-24T11:20:17.727165+0000","last_clean":"2024-03-24T11:20:17.727165+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:20:17.727165+0000","last_undegraded":"2024-03-24T11:20:17.727165+0000","last_fullsized":"2024-03-24T11:20:17.727165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725848+0000","last_change":"2024-03-24T11:20:16.429420+0000","last_active":"2024-03-24T11:20:17.725848+0000","last_peered":"2024-03-24T11:20:17.725848+0000","last_clean":"2024-03-24T11:20:17.725848+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:20:17.725848+0000","last_undegraded":"2024-03-24T11:20:17.725848+0000","last_fullsized":"2024-03-24T11:20:17.725848+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:16.429342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:16.429342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.285148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727699+0000","last_change":"2024-03-24T11:18:56.141810+0000","last_active":"2024-03-24T11:20:17.727699+0000","last_peered":"2024-03-24T11:20:17.727699+0000","last_clean":"2024-03-24T11:20:17.727699+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:20:17.727699+0000","last_undegraded":"2024-03-24T11:20:17.727699+0000","last_fullsized":"2024-03-24T11:20:17.727699+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720134+0000","last_change":"2024-03-24T11:18:59.714114+0000","last_active":"2024-03-24T11:20:17.720134+0000","last_peered":"2024-03-24T11:20:17.720134+0000","last_clean":"2024-03-24T11:20:17.720134+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:20:17.720134+0000","last_undegraded":"2024-03-24T11:20:17.720134+0000","last_fullsized":"2024-03-24T11:20:17.720134+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.411365+0000","last_change":"2024-03-24T11:20:20.411389+0000","last_active":"2024-03-24T11:20:20.411365+0000","last_peered":"2024-03-24T11:20:20.411365+0000","last_clean":"2024-03-24T11:20:20.411365+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:20:20.411365+0000","last_undegraded":"2024-03-24T11:20:20.411365+0000","last_fullsized":"2024-03-24T11:20:20.411365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:20.411287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.411287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.233362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726215+0000","last_change":"2024-03-24T11:18:53.681700+0000","last_active":"2024-03-24T11:20:17.726215+0000","last_peered":"2024-03-24T11:20:17.726215+0000","last_clean":"2024-03-24T11:20:17.726215+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:20:17.726215+0000","last_undegraded":"2024-03-24T11:20:17.726215+0000","last_fullsized":"2024-03-24T11:20:17.726215+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.896550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722390+0000","last_change":"2024-03-24T11:18:55.683808+0000","last_active":"2024-03-24T11:20:17.722390+0000","last_peered":"2024-03-24T11:20:17.722390+0000","last_clean":"2024-03-24T11:20:17.722390+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:20:17.722390+0000","last_undegraded":"2024-03-24T11:20:17.722390+0000","last_fullsized":"2024-03-24T11:20:17.722390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727515+0000","last_change":"2024-03-24T11:20:03.441868+0000","last_active":"2024-03-24T11:20:17.727515+0000","last_peered":"2024-03-24T11:20:17.727515+0000","last_clean":"2024-03-24T11:20:17.727515+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:20:17.727515+0000","last_undegraded":"2024-03-24T11:20:17.727515+0000","last_fullsized":"2024-03-24T11:20:17.727515+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.441791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.441791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.538293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720795+0000","last_change":"2024-03-24T11:18:57.703012+0000","last_active":"2024-03-24T11:20:17.720795+0000","last_peered":"2024-03-24T11:20:17.720795+0000","last_clean":"2024-03-24T11:20:17.720795+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:20:17.720795+0000","last_undegraded":"2024-03-24T11:20:17.720795+0000","last_fullsized":"2024-03-24T11:20:17.720795+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.320906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718306+0000","last_change":"2024-03-24T11:19:56.100757+0000","last_active":"2024-03-24T11:20:17.718306+0000","last_peered":"2024-03-24T11:20:17.718306+0000","last_clean":"2024-03-24T11:20:17.718306+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:20:17.718306+0000","last_undegraded":"2024-03-24T11:20:17.718306+0000","last_fullsized":"2024-03-24T11:20:17.718306+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.100633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.072386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718274+0000","last_change":"2024-03-24T11:20:04.036645+0000","last_active":"2024-03-24T11:20:17.718274+0000","last_peered":"2024-03-24T11:20:17.718274+0000","last_clean":"2024-03-24T11:20:17.718274+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:20:17.718274+0000","last_undegraded":"2024-03-24T11:20:17.718274+0000","last_fullsized":"2024-03-24T11:20:17.718274+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:20:04.036510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.036510+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.183695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721407+0000","last_change":"2024-03-24T11:18:59.717539+0000","last_active":"2024-03-24T11:20:17.721407+0000","last_peered":"2024-03-24T11:20:17.721407+0000","last_clean":"2024-03-24T11:20:17.721407+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:20:17.721407+0000","last_undegraded":"2024-03-24T11:20:17.721407+0000","last_fullsized":"2024-03-24T11:20:17.721407+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.956051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725276+0000","last_change":"2024-03-24T11:20:07.390710+0000","last_active":"2024-03-24T11:20:17.725276+0000","last_peered":"2024-03-24T11:20:17.725276+0000","last_clean":"2024-03-24T11:20:17.725276+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:20:17.725276+0000","last_undegraded":"2024-03-24T11:20:17.725276+0000","last_fullsized":"2024-03-24T11:20:17.725276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:07.390604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.390604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.892391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725427+0000","last_change":"2024-03-24T11:18:53.682165+0000","last_active":"2024-03-24T11:20:17.725427+0000","last_peered":"2024-03-24T11:20:17.725427+0000","last_clean":"2024-03-24T11:20:17.725427+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:20:17.725427+0000","last_undegraded":"2024-03-24T11:20:17.725427+0000","last_fullsized":"2024-03-24T11:20:17.725427+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.317744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728228+0000","last_change":"2024-03-24T11:18:56.141308+0000","last_active":"2024-03-24T11:20:17.728228+0000","last_peered":"2024-03-24T11:20:17.728228+0000","last_clean":"2024-03-24T11:20:17.728228+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:20:17.728228+0000","last_undegraded":"2024-03-24T11:20:17.728228+0000","last_fullsized":"2024-03-24T11:20:17.728228+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718789+0000","last_change":"2024-03-24T11:18:59.705914+0000","last_active":"2024-03-24T11:20:17.718789+0000","last_peered":"2024-03-24T11:20:17.718789+0000","last_clean":"2024-03-24T11:20:17.718789+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:20:17.718789+0000","last_undegraded":"2024-03-24T11:20:17.718789+0000","last_fullsized":"2024-03-24T11:20:17.718789+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728416+0000","last_change":"2024-03-24T11:20:04.436584+0000","last_active":"2024-03-24T11:20:17.728416+0000","last_peered":"2024-03-24T11:20:17.728416+0000","last_clean":"2024-03-24T11:20:17.728416+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:20:17.728416+0000","last_undegraded":"2024-03-24T11:20:17.728416+0000","last_fullsized":"2024-03-24T11:20:17.728416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:04.436493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.436493+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.892034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721104+0000","last_change":"2024-03-24T11:18:53.683017+0000","last_active":"2024-03-24T11:20:17.721104+0000","last_peered":"2024-03-24T11:20:17.721104+0000","last_clean":"2024-03-24T11:20:17.721104+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:20:17.721104+0000","last_undegraded":"2024-03-24T11:20:17.721104+0000","last_fullsized":"2024-03-24T11:20:17.721104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725463+0000","last_change":"2024-03-24T11:18:56.143374+0000","last_active":"2024-03-24T11:20:17.725463+0000","last_peered":"2024-03-24T11:20:17.725463+0000","last_clean":"2024-03-24T11:20:17.725463+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:20:17.725463+0000","last_undegraded":"2024-03-24T11:20:17.725463+0000","last_fullsized":"2024-03-24T11:20:17.725463+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:21.462286+0000","last_change":"2024-03-24T11:20:21.462307+0000","last_active":"2024-03-24T11:20:21.462286+0000","last_peered":"2024-03-24T11:20:21.462286+0000","last_clean":"2024-03-24T11:20:21.462286+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:20:21.462286+0000","last_undegraded":"2024-03-24T11:20:21.462286+0000","last_fullsized":"2024-03-24T11:20:21.462286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:21.462219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:21.462219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.223620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728652+0000","last_change":"2024-03-24T11:18:57.694184+0000","last_active":"2024-03-24T11:20:17.728652+0000","last_peered":"2024-03-24T11:20:17.728652+0000","last_clean":"2024-03-24T11:20:17.728652+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:20:17.728652+0000","last_undegraded":"2024-03-24T11:20:17.728652+0000","last_fullsized":"2024-03-24T11:20:17.728652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725168+0000","last_change":"2024-03-24T11:19:58.799434+0000","last_active":"2024-03-24T11:20:17.725168+0000","last_peered":"2024-03-24T11:20:17.725168+0000","last_clean":"2024-03-24T11:20:17.725168+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:20:17.725168+0000","last_undegraded":"2024-03-24T11:20:17.725168+0000","last_fullsized":"2024-03-24T11:20:17.725168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.799295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.799295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.674050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721944+0000","last_change":"2024-03-24T11:19:58.001386+0000","last_active":"2024-03-24T11:20:17.721944+0000","last_peered":"2024-03-24T11:20:17.721944+0000","last_clean":"2024-03-24T11:20:17.721944+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:20:17.721944+0000","last_undegraded":"2024-03-24T11:20:17.721944+0000","last_fullsized":"2024-03-24T11:20:17.721944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.001296+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.932223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726170+0000","last_change":"2024-03-24T11:19:00.314565+0000","last_active":"2024-03-24T11:20:17.726170+0000","last_peered":"2024-03-24T11:20:17.726170+0000","last_clean":"2024-03-24T11:20:17.726170+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:20:17.726170+0000","last_undegraded":"2024-03-24T11:20:17.726170+0000","last_fullsized":"2024-03-24T11:20:17.726170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.091963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725927+0000","last_change":"2024-03-24T11:18:57.693993+0000","last_active":"2024-03-24T11:20:17.725927+0000","last_peered":"2024-03-24T11:20:17.725927+0000","last_clean":"2024-03-24T11:20:17.725927+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:20:17.725927+0000","last_undegraded":"2024-03-24T11:20:17.725927+0000","last_fullsized":"2024-03-24T11:20:17.725927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.747158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725214+0000","last_change":"2024-03-24T11:20:01.231696+0000","last_active":"2024-03-24T11:20:17.725214+0000","last_peered":"2024-03-24T11:20:17.725214+0000","last_clean":"2024-03-24T11:20:17.725214+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:20:17.725214+0000","last_undegraded":"2024-03-24T11:20:17.725214+0000","last_fullsized":"2024-03-24T11:20:17.725214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:01.231617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.231617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.010965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":60,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728656+0000","last_change":"2024-03-24T11:18:56.142569+0000","last_active":"2024-03-24T11:20:17.728656+0000","last_peered":"2024-03-24T11:20:17.728656+0000","last_clean":"2024-03-24T11:20:17.728656+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:20:17.728656+0000","last_undegraded":"2024-03-24T11:20:17.728656+0000","last_fullsized":"2024-03-24T11:20:17.728656+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.660826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724852+0000","last_change":"2024-03-24T11:20:02.408219+0000","last_active":"2024-03-24T11:20:17.724852+0000","last_peered":"2024-03-24T11:20:17.724852+0000","last_clean":"2024-03-24T11:20:17.724852+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:20:17.724852+0000","last_undegraded":"2024-03-24T11:20:17.724852+0000","last_fullsized":"2024-03-24T11:20:17.724852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.408130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.408130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:03.752282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726601+0000","last_change":"2024-03-24T11:20:15.827098+0000","last_active":"2024-03-24T11:20:17.726601+0000","last_peered":"2024-03-24T11:20:17.726601+0000","last_clean":"2024-03-24T11:20:17.726601+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:20:17.726601+0000","last_undegraded":"2024-03-24T11:20:17.726601+0000","last_fullsized":"2024-03-24T11:20:17.726601+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-24T11:20:15.827014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:15.827014+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:26.060294+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724471+0000","last_change":"2024-03-24T11:20:12.354666+0000","last_active":"2024-03-24T11:20:17.724471+0000","last_peered":"2024-03-24T11:20:17.724471+0000","last_clean":"2024-03-24T11:20:17.724471+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:20:17.724471+0000","last_undegraded":"2024-03-24T11:20:17.724471+0000","last_fullsized":"2024-03-24T11:20:17.724471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:20:12.354559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.354559+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.930978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725443+0000","last_change":"2024-03-24T11:20:05.863734+0000","last_active":"2024-03-24T11:20:17.725443+0000","last_peered":"2024-03-24T11:20:17.725443+0000","last_clean":"2024-03-24T11:20:17.725443+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:20:17.725443+0000","last_undegraded":"2024-03-24T11:20:17.725443+0000","last_fullsized":"2024-03-24T11:20:17.725443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:05.863616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.863616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:09.078050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727862+0000","last_change":"2024-03-24T11:18:59.706989+0000","last_active":"2024-03-24T11:20:17.727862+0000","last_peered":"2024-03-24T11:20:17.727862+0000","last_clean":"2024-03-24T11:20:17.727862+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:20:17.727862+0000","last_undegraded":"2024-03-24T11:20:17.727862+0000","last_fullsized":"2024-03-24T11:20:17.727862+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726606+0000","last_change":"2024-03-24T11:18:53.682847+0000","last_active":"2024-03-24T11:20:17.726606+0000","last_peered":"2024-03-24T11:20:17.726606+0000","last_clean":"2024-03-24T11:20:17.726606+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:20:17.726606+0000","last_undegraded":"2024-03-24T11:20:17.726606+0000","last_fullsized":"2024-03-24T11:20:17.726606+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.562909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.132832+0000","last_change":"2024-03-24T11:20:20.132861+0000","last_active":"2024-03-24T11:20:20.132832+0000","last_peered":"2024-03-24T11:20:20.132832+0000","last_clean":"2024-03-24T11:20:20.132832+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:20:20.132832+0000","last_undegraded":"2024-03-24T11:20:20.132832+0000","last_fullsized":"2024-03-24T11:20:20.132832+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:20:20.132772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.132772+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:40.124937+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724647+0000","last_change":"2024-03-24T11:18:57.704315+0000","last_active":"2024-03-24T11:20:17.724647+0000","last_peered":"2024-03-24T11:20:17.724647+0000","last_clean":"2024-03-24T11:20:17.724647+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:20:17.724647+0000","last_undegraded":"2024-03-24T11:20:17.724647+0000","last_fullsized":"2024-03-24T11:20:17.724647+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.986639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726889+0000","last_change":"2024-03-24T11:18:59.707271+0000","last_active":"2024-03-24T11:20:17.726889+0000","last_peered":"2024-03-24T11:20:17.726889+0000","last_clean":"2024-03-24T11:20:17.726889+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:20:17.726889+0000","last_undegraded":"2024-03-24T11:20:17.726889+0000","last_fullsized":"2024-03-24T11:20:17.726889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.298674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.474333+0000","last_change":"2024-03-24T11:20:20.474352+0000","last_active":"2024-03-24T11:20:20.474333+0000","last_peered":"2024-03-24T11:20:20.474333+0000","last_clean":"2024-03-24T11:20:20.474333+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:20:20.474333+0000","last_undegraded":"2024-03-24T11:20:20.474333+0000","last_fullsized":"2024-03-24T11:20:20.474333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:20.474279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.474279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:33.479751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726315+0000","last_change":"2024-03-24T11:20:00.851820+0000","last_active":"2024-03-24T11:20:17.726315+0000","last_peered":"2024-03-24T11:20:17.726315+0000","last_clean":"2024-03-24T11:20:17.726315+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:20:17.726315+0000","last_undegraded":"2024-03-24T11:20:17.726315+0000","last_fullsized":"2024-03-24T11:20:17.726315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.851691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.851691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.342041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726280+0000","last_change":"2024-03-24T11:18:56.143415+0000","last_active":"2024-03-24T11:20:17.726280+0000","last_peered":"2024-03-24T11:20:17.726280+0000","last_clean":"2024-03-24T11:20:17.726280+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:20:17.726280+0000","last_undegraded":"2024-03-24T11:20:17.726280+0000","last_fullsized":"2024-03-24T11:20:17.726280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.105644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720393+0000","last_change":"2024-03-24T11:20:07.980193+0000","last_active":"2024-03-24T11:20:17.720393+0000","last_peered":"2024-03-24T11:20:17.720393+0000","last_clean":"2024-03-24T11:20:17.720393+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:20:17.720393+0000","last_undegraded":"2024-03-24T11:20:17.720393+0000","last_fullsized":"2024-03-24T11:20:17.720393+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.980113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.980113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.058894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724832+0000","last_change":"2024-03-24T11:20:13.558725+0000","last_active":"2024-03-24T11:20:17.724832+0000","last_peered":"2024-03-24T11:20:17.724832+0000","last_clean":"2024-03-24T11:20:17.724832+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:20:17.724832+0000","last_undegraded":"2024-03-24T11:20:17.724832+0000","last_fullsized":"2024-03-24T11:20:17.724832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:13.558645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:13.558645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:23.034282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724808+0000","last_change":"2024-03-24T11:19:56.404250+0000","last_active":"2024-03-24T11:20:17.724808+0000","last_peered":"2024-03-24T11:20:17.724808+0000","last_clean":"2024-03-24T11:20:17.724808+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:20:17.724808+0000","last_undegraded":"2024-03-24T11:20:17.724808+0000","last_fullsized":"2024-03-24T11:20:17.724808+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:19:56.404146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.404146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.193173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721637+0000","last_change":"2024-03-24T11:18:55.684166+0000","last_active":"2024-03-24T11:20:17.721637+0000","last_peered":"2024-03-24T11:20:17.721637+0000","last_clean":"2024-03-24T11:20:17.721637+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:20:17.721637+0000","last_undegraded":"2024-03-24T11:20:17.721637+0000","last_fullsized":"2024-03-24T11:20:17.721637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:19.368389+0000","last_change":"2024-03-24T11:20:19.368409+0000","last_active":"2024-03-24T11:20:19.368389+0000","last_peered":"2024-03-24T11:20:19.368389+0000","last_clean":"2024-03-24T11:20:19.368389+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:20:19.368389+0000","last_undegraded":"2024-03-24T11:20:19.368389+0000","last_fullsized":"2024-03-24T11:20:19.368389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:19.368309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:19.368309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:42.640854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724564+0000","last_change":"2024-03-24T11:18:56.680518+0000","last_active":"2024-03-24T11:20:17.724564+0000","last_peered":"2024-03-24T11:20:17.724564+0000","last_clean":"2024-03-24T11:20:17.724564+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:20:17.724564+0000","last_undegraded":"2024-03-24T11:20:17.724564+0000","last_fullsized":"2024-03-24T11:20:17.724564+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:18:51.653668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718111+0000","last_change":"2024-03-24T11:18:53.680619+0000","last_active":"2024-03-24T11:20:17.718111+0000","last_peered":"2024-03-24T11:20:17.718111+0000","last_clean":"2024-03-24T11:20:17.718111+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:20:17.718111+0000","last_undegraded":"2024-03-24T11:20:17.718111+0000","last_fullsized":"2024-03-24T11:20:17.718111+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":125,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722368+0000","last_change":"2024-03-24T11:19:55.086981+0000","last_active":"2024-03-24T11:20:17.722368+0000","last_peered":"2024-03-24T11:20:17.722368+0000","last_clean":"2024-03-24T11:20:17.722368+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:20:17.722368+0000","last_undegraded":"2024-03-24T11:20:17.722368+0000","last_fullsized":"2024-03-24T11:20:17.722368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:19:55.086847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:19:55.086847+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:14.068671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720596+0000","last_change":"2024-03-24T11:18:57.694386+0000","last_active":"2024-03-24T11:20:17.720596+0000","last_peered":"2024-03-24T11:20:17.720596+0000","last_clean":"2024-03-24T11:20:17.720596+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:20:17.720596+0000","last_undegraded":"2024-03-24T11:20:17.720596+0000","last_fullsized":"2024-03-24T11:20:17.720596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726397+0000","last_change":"2024-03-24T11:20:12.833645+0000","last_active":"2024-03-24T11:20:17.726397+0000","last_peered":"2024-03-24T11:20:17.726397+0000","last_clean":"2024-03-24T11:20:17.726397+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:20:17.726397+0000","last_undegraded":"2024-03-24T11:20:17.726397+0000","last_fullsized":"2024-03-24T11:20:17.726397+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:12.833509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.833509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.607503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721577+0000","last_change":"2024-03-24T11:19:58.185937+0000","last_active":"2024-03-24T11:20:17.721577+0000","last_peered":"2024-03-24T11:20:17.721577+0000","last_clean":"2024-03-24T11:20:17.721577+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:20:17.721577+0000","last_undegraded":"2024-03-24T11:20:17.721577+0000","last_fullsized":"2024-03-24T11:20:17.721577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.185832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.185832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.274032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724784+0000","last_change":"2024-03-24T11:20:00.555990+0000","last_active":"2024-03-24T11:20:17.724784+0000","last_peered":"2024-03-24T11:20:17.724784+0000","last_clean":"2024-03-24T11:20:17.724784+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:20:17.724784+0000","last_undegraded":"2024-03-24T11:20:17.724784+0000","last_fullsized":"2024-03-24T11:20:17.724784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.555913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.555913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.222390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724932+0000","last_change":"2024-03-24T11:20:02.552338+0000","last_active":"2024-03-24T11:20:17.724932+0000","last_peered":"2024-03-24T11:20:17.724932+0000","last_clean":"2024-03-24T11:20:17.724932+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:20:17.724932+0000","last_undegraded":"2024-03-24T11:20:17.724932+0000","last_fullsized":"2024-03-24T11:20:17.724932+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:20:02.552224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.552224+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.289260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725112+0000","last_change":"2024-03-24T11:20:08.530821+0000","last_active":"2024-03-24T11:20:17.725112+0000","last_peered":"2024-03-24T11:20:17.725112+0000","last_clean":"2024-03-24T11:20:17.725112+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:20:17.725112+0000","last_undegraded":"2024-03-24T11:20:17.725112+0000","last_fullsized":"2024-03-24T11:20:17.725112+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:08.530740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:08.530740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.295185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726240+0000","last_change":"2024-03-24T11:20:17.382742+0000","last_active":"2024-03-24T11:20:17.726240+0000","last_peered":"2024-03-24T11:20:17.726240+0000","last_clean":"2024-03-24T11:20:17.726240+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:20:17.726240+0000","last_undegraded":"2024-03-24T11:20:17.726240+0000","last_fullsized":"2024-03-24T11:20:17.726240+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:17.382613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:17.382613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.038316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":308,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726982+0000","last_change":"2024-03-24T11:19:18.724688+0000","last_active":"2024-03-24T11:20:17.726982+0000","last_peered":"2024-03-24T11:20:17.726982+0000","last_clean":"2024-03-24T11:20:17.726982+0000","last_became_active":"2024-03-24T11:18:44.123994+0000","last_became_peered":"2024-03-24T11:18:44.123994+0000","last_unstale":"2024-03-24T11:20:17.726982+0000","last_undegraded":"2024-03-24T11:20:17.726982+0000","last_fullsized":"2024-03-24T11:20:17.726982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-24T11:19:18.724552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:19:18.724552+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:46.099214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727873+0000","last_change":"2024-03-24T11:20:05.392270+0000","last_active":"2024-03-24T11:20:17.727873+0000","last_peered":"2024-03-24T11:20:17.727873+0000","last_clean":"2024-03-24T11:20:17.727873+0000","last_became_active":"2024-03-24T11:18:53.678099+0000","last_became_peered":"2024-03-24T11:18:53.678099+0000","last_unstale":"2024-03-24T11:20:17.727873+0000","last_undegraded":"2024-03-24T11:20:17.727873+0000","last_fullsized":"2024-03-24T11:20:17.727873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.392195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.392195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.644580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727438+0000","last_change":"2024-03-24T11:18:56.141576+0000","last_active":"2024-03-24T11:20:17.727438+0000","last_peered":"2024-03-24T11:20:17.727438+0000","last_clean":"2024-03-24T11:20:17.727438+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:20:17.727438+0000","last_undegraded":"2024-03-24T11:20:17.727438+0000","last_fullsized":"2024-03-24T11:20:17.727438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.491233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725997+0000","last_change":"2024-03-24T11:20:10.834643+0000","last_active":"2024-03-24T11:20:17.725997+0000","last_peered":"2024-03-24T11:20:17.725997+0000","last_clean":"2024-03-24T11:20:17.725997+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:20:17.725997+0000","last_undegraded":"2024-03-24T11:20:17.725997+0000","last_fullsized":"2024-03-24T11:20:17.725997+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:10.834537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:10.834537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:23.214600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721802+0000","last_change":"2024-03-24T11:18:59.710726+0000","last_active":"2024-03-24T11:20:17.721802+0000","last_peered":"2024-03-24T11:20:17.721802+0000","last_clean":"2024-03-24T11:20:17.721802+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:20:17.721802+0000","last_undegraded":"2024-03-24T11:20:17.721802+0000","last_fullsized":"2024-03-24T11:20:17.721802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.585199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720709+0000","last_change":"2024-03-24T11:18:53.681515+0000","last_active":"2024-03-24T11:20:17.720709+0000","last_peered":"2024-03-24T11:20:17.720709+0000","last_clean":"2024-03-24T11:20:17.720709+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:20:17.720709+0000","last_undegraded":"2024-03-24T11:20:17.720709+0000","last_fullsized":"2024-03-24T11:20:17.720709+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":158,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:21.614060+0000","last_change":"2024-03-24T11:20:06.880339+0000","last_active":"2024-03-24T11:20:21.614060+0000","last_peered":"2024-03-24T11:20:21.614060+0000","last_clean":"2024-03-24T11:20:21.614060+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:20:21.614060+0000","last_undegraded":"2024-03-24T11:20:21.614060+0000","last_fullsized":"2024-03-24T11:20:21.614060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:20:06.880224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.880224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:32.826706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721534+0000","last_change":"2024-03-24T11:18:55.681576+0000","last_active":"2024-03-24T11:20:17.721534+0000","last_peered":"2024-03-24T11:20:17.721534+0000","last_clean":"2024-03-24T11:20:17.721534+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:20:17.721534+0000","last_undegraded":"2024-03-24T11:20:17.721534+0000","last_fullsized":"2024-03-24T11:20:17.721534+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.955414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726622+0000","last_change":"2024-03-24T11:20:13.385604+0000","last_active":"2024-03-24T11:20:17.726622+0000","last_peered":"2024-03-24T11:20:17.726622+0000","last_clean":"2024-03-24T11:20:17.726622+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:20:17.726622+0000","last_undegraded":"2024-03-24T11:20:17.726622+0000","last_fullsized":"2024-03-24T11:20:17.726622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:13.385497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:13.385497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.985789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722118+0000","last_change":"2024-03-24T11:20:17.347069+0000","last_active":"2024-03-24T11:20:17.722118+0000","last_peered":"2024-03-24T11:20:17.722118+0000","last_clean":"2024-03-24T11:20:17.722118+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:20:17.722118+0000","last_undegraded":"2024-03-24T11:20:17.722118+0000","last_fullsized":"2024-03-24T11:20:17.722118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:17.346965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:17.346965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:44.870753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722018+0000","last_change":"2024-03-24T11:20:07.202667+0000","last_active":"2024-03-24T11:20:17.722018+0000","last_peered":"2024-03-24T11:20:17.722018+0000","last_clean":"2024-03-24T11:20:17.722018+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:20:17.722018+0000","last_undegraded":"2024-03-24T11:20:17.722018+0000","last_fullsized":"2024-03-24T11:20:17.722018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.202576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.202576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:10.625674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":87,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721344+0000","last_change":"2024-03-24T11:18:56.142025+0000","last_active":"2024-03-24T11:20:17.721344+0000","last_peered":"2024-03-24T11:20:17.721344+0000","last_clean":"2024-03-24T11:20:17.721344+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:20:17.721344+0000","last_undegraded":"2024-03-24T11:20:17.721344+0000","last_fullsized":"2024-03-24T11:20:17.721344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.374097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726350+0000","last_change":"2024-03-24T11:20:05.186178+0000","last_active":"2024-03-24T11:20:17.726350+0000","last_peered":"2024-03-24T11:20:17.726350+0000","last_clean":"2024-03-24T11:20:17.726350+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:20:17.726350+0000","last_undegraded":"2024-03-24T11:20:17.726350+0000","last_fullsized":"2024-03-24T11:20:17.726350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.186078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.695962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720276+0000","last_change":"2024-03-24T11:18:59.714865+0000","last_active":"2024-03-24T11:20:17.720276+0000","last_peered":"2024-03-24T11:20:17.720276+0000","last_clean":"2024-03-24T11:20:17.720276+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:20:17.720276+0000","last_undegraded":"2024-03-24T11:20:17.720276+0000","last_fullsized":"2024-03-24T11:20:17.720276+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725326+0000","last_change":"2024-03-24T11:20:09.320186+0000","last_active":"2024-03-24T11:20:17.725326+0000","last_peered":"2024-03-24T11:20:17.725326+0000","last_clean":"2024-03-24T11:20:17.725326+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:20:17.725326+0000","last_undegraded":"2024-03-24T11:20:17.725326+0000","last_fullsized":"2024-03-24T11:20:17.725326+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:09.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:09.320059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:11.542154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720896+0000","last_change":"2024-03-24T11:20:14.364904+0000","last_active":"2024-03-24T11:20:17.720896+0000","last_peered":"2024-03-24T11:20:17.720896+0000","last_clean":"2024-03-24T11:20:17.720896+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:20:17.720896+0000","last_undegraded":"2024-03-24T11:20:17.720896+0000","last_fullsized":"2024-03-24T11:20:17.720896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:20:14.364738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:14.364738+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:15.151256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721797+0000","last_change":"2024-03-24T11:20:01.039976+0000","last_active":"2024-03-24T11:20:17.721797+0000","last_peered":"2024-03-24T11:20:17.721797+0000","last_clean":"2024-03-24T11:20:17.721797+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:20:17.721797+0000","last_undegraded":"2024-03-24T11:20:17.721797+0000","last_fullsized":"2024-03-24T11:20:17.721797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:01.039839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.039839+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:01.168501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725621+0000","last_change":"2024-03-24T11:19:00.314552+0000","last_active":"2024-03-24T11:20:17.725621+0000","last_peered":"2024-03-24T11:20:17.725621+0000","last_clean":"2024-03-24T11:20:17.725621+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:20:17.725621+0000","last_undegraded":"2024-03-24T11:20:17.725621+0000","last_fullsized":"2024-03-24T11:20:17.725621+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721666+0000","last_change":"2024-03-24T11:18:53.683466+0000","last_active":"2024-03-24T11:20:17.721666+0000","last_peered":"2024-03-24T11:20:17.721666+0000","last_clean":"2024-03-24T11:20:17.721666+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:20:17.721666+0000","last_undegraded":"2024-03-24T11:20:17.721666+0000","last_fullsized":"2024-03-24T11:20:17.721666+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717808+0000","last_change":"2024-03-24T11:20:14.019874+0000","last_active":"2024-03-24T11:20:17.717808+0000","last_peered":"2024-03-24T11:20:17.717808+0000","last_clean":"2024-03-24T11:20:17.717808+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:20:17.717808+0000","last_undegraded":"2024-03-24T11:20:17.717808+0000","last_fullsized":"2024-03-24T11:20:17.717808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-03-24T11:20:14.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:14.019717+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:42.126712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727799+0000","last_change":"2024-03-24T11:18:57.697565+0000","last_active":"2024-03-24T11:20:17.727799+0000","last_peered":"2024-03-24T11:20:17.727799+0000","last_clean":"2024-03-24T11:20:17.727799+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:20:17.727799+0000","last_undegraded":"2024-03-24T11:20:17.727799+0000","last_fullsized":"2024-03-24T11:20:17.727799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.519845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717952+0000","last_change":"2024-03-24T11:18:59.706057+0000","last_active":"2024-03-24T11:20:17.717952+0000","last_peered":"2024-03-24T11:20:17.717952+0000","last_clean":"2024-03-24T11:20:17.717952+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:20:17.717952+0000","last_undegraded":"2024-03-24T11:20:17.717952+0000","last_fullsized":"2024-03-24T11:20:17.717952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.016361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728503+0000","last_change":"2024-03-24T11:18:53.679203+0000","last_active":"2024-03-24T11:20:17.728503+0000","last_peered":"2024-03-24T11:20:17.728503+0000","last_clean":"2024-03-24T11:20:17.728503+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:20:17.728503+0000","last_undegraded":"2024-03-24T11:20:17.728503+0000","last_fullsized":"2024-03-24T11:20:17.728503+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":83,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727648+0000","last_change":"2024-03-24T11:18:56.141573+0000","last_active":"2024-03-24T11:20:17.727648+0000","last_peered":"2024-03-24T11:20:17.727648+0000","last_clean":"2024-03-24T11:20:17.727648+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:20:17.727648+0000","last_undegraded":"2024-03-24T11:20:17.727648+0000","last_fullsized":"2024-03-24T11:20:17.727648+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.850624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717620+0000","last_change":"2024-03-24T11:20:00.029193+0000","last_active":"2024-03-24T11:20:17.717620+0000","last_peered":"2024-03-24T11:20:17.717620+0000","last_clean":"2024-03-24T11:20:17.717620+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:20:17.717620+0000","last_undegraded":"2024-03-24T11:20:17.717620+0000","last_fullsized":"2024-03-24T11:20:17.717620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.029091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.029091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.480294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725049+0000","last_change":"2024-03-24T11:18:59.715347+0000","last_active":"2024-03-24T11:20:17.725049+0000","last_peered":"2024-03-24T11:20:17.725049+0000","last_clean":"2024-03-24T11:20:17.725049+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:20:17.725049+0000","last_undegraded":"2024-03-24T11:20:17.725049+0000","last_fullsized":"2024-03-24T11:20:17.725049+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.582227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718824+0000","last_change":"2024-03-24T11:18:57.699380+0000","last_active":"2024-03-24T11:20:17.718824+0000","last_peered":"2024-03-24T11:20:17.718824+0000","last_clean":"2024-03-24T11:20:17.718824+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:20:17.718824+0000","last_undegraded":"2024-03-24T11:20:17.718824+0000","last_fullsized":"2024-03-24T11:20:17.718824+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727641+0000","last_change":"2024-03-24T11:20:02.414997+0000","last_active":"2024-03-24T11:20:17.727641+0000","last_peered":"2024-03-24T11:20:17.727641+0000","last_clean":"2024-03-24T11:20:17.727641+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:20:17.727641+0000","last_undegraded":"2024-03-24T11:20:17.727641+0000","last_fullsized":"2024-03-24T11:20:17.727641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.414845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.414845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.613098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722633+0000","last_change":"2024-03-24T11:18:55.686410+0000","last_active":"2024-03-24T11:20:17.722633+0000","last_peered":"2024-03-24T11:20:17.722633+0000","last_clean":"2024-03-24T11:20:17.722633+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:20:17.722633+0000","last_undegraded":"2024-03-24T11:20:17.722633+0000","last_fullsized":"2024-03-24T11:20:17.722633+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727990+0000","last_change":"2024-03-24T11:19:00.312496+0000","last_active":"2024-03-24T11:20:17.727990+0000","last_peered":"2024-03-24T11:20:17.727990+0000","last_clean":"2024-03-24T11:20:17.727990+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:20:17.727990+0000","last_undegraded":"2024-03-24T11:20:17.727990+0000","last_fullsized":"2024-03-24T11:20:17.727990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726542+0000","last_change":"2024-03-24T11:20:15.381720+0000","last_active":"2024-03-24T11:20:17.726542+0000","last_peered":"2024-03-24T11:20:17.726542+0000","last_clean":"2024-03-24T11:20:17.726542+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:20:17.726542+0000","last_undegraded":"2024-03-24T11:20:17.726542+0000","last_fullsized":"2024-03-24T11:20:17.726542+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:15.381636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:15.381636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:32.306821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717707+0000","last_change":"2024-03-24T11:18:53.679465+0000","last_active":"2024-03-24T11:20:17.717707+0000","last_peered":"2024-03-24T11:20:17.717707+0000","last_clean":"2024-03-24T11:20:17.717707+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:20:17.717707+0000","last_undegraded":"2024-03-24T11:20:17.717707+0000","last_fullsized":"2024-03-24T11:20:17.717707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.456813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:19.106711+0000","last_change":"2024-03-24T11:20:19.106754+0000","last_active":"2024-03-24T11:20:19.106711+0000","last_peered":"2024-03-24T11:20:19.106711+0000","last_clean":"2024-03-24T11:20:19.106711+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:20:19.106711+0000","last_undegraded":"2024-03-24T11:20:19.106711+0000","last_fullsized":"2024-03-24T11:20:19.106711+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:20:19.106649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:19.106649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:38.089530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725356+0000","last_change":"2024-03-24T11:20:06.548123+0000","last_active":"2024-03-24T11:20:17.725356+0000","last_peered":"2024-03-24T11:20:17.725356+0000","last_clean":"2024-03-24T11:20:17.725356+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:20:17.725356+0000","last_undegraded":"2024-03-24T11:20:17.725356+0000","last_fullsized":"2024-03-24T11:20:17.725356+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:06.548036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.548036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:08.537924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722247+0000","last_change":"2024-03-24T11:20:04.159022+0000","last_active":"2024-03-24T11:20:17.722247+0000","last_peered":"2024-03-24T11:20:17.722247+0000","last_clean":"2024-03-24T11:20:17.722247+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:20:17.722247+0000","last_undegraded":"2024-03-24T11:20:17.722247+0000","last_fullsized":"2024-03-24T11:20:17.722247+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:04.158945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.158945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.212380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726123+0000","last_change":"2024-03-24T11:20:03.233345+0000","last_active":"2024-03-24T11:20:17.726123+0000","last_peered":"2024-03-24T11:20:17.726123+0000","last_clean":"2024-03-24T11:20:17.726123+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:20:17.726123+0000","last_undegraded":"2024-03-24T11:20:17.726123+0000","last_fullsized":"2024-03-24T11:20:17.726123+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:03.233200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.233200+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.734595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":103,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718574+0000","last_change":"2024-03-24T11:19:18.829942+0000","last_active":"2024-03-24T11:20:17.718574+0000","last_peered":"2024-03-24T11:20:17.718574+0000","last_clean":"2024-03-24T11:20:17.718574+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:20:17.718574+0000","last_undegraded":"2024-03-24T11:20:17.718574+0000","last_fullsized":"2024-03-24T11:20:17.718574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718823+0000","last_change":"2024-03-24T11:20:16.026203+0000","last_active":"2024-03-24T11:20:17.718823+0000","last_peered":"2024-03-24T11:20:17.718823+0000","last_clean":"2024-03-24T11:20:17.718823+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:20:17.718823+0000","last_undegraded":"2024-03-24T11:20:17.718823+0000","last_fullsized":"2024-03-24T11:20:17.718823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:20:16.026113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:38.997969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725137+0000","last_change":"2024-03-24T11:18:57.701581+0000","last_active":"2024-03-24T11:20:17.725137+0000","last_peered":"2024-03-24T11:20:17.725137+0000","last_clean":"2024-03-24T11:20:17.725137+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:20:17.725137+0000","last_undegraded":"2024-03-24T11:20:17.725137+0000","last_fullsized":"2024-03-24T11:20:17.725137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728023+0000","last_change":"2024-03-24T11:18:53.685545+0000","last_active":"2024-03-24T11:20:17.728023+0000","last_peered":"2024-03-24T11:20:17.728023+0000","last_clean":"2024-03-24T11:20:17.728023+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:20:17.728023+0000","last_undegraded":"2024-03-24T11:20:17.728023+0000","last_fullsized":"2024-03-24T11:20:17.728023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718394+0000","last_change":"2024-03-24T11:18:55.680886+0000","last_active":"2024-03-24T11:20:17.718394+0000","last_peered":"2024-03-24T11:20:17.718394+0000","last_clean":"2024-03-24T11:20:17.718394+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:20:17.718394+0000","last_undegraded":"2024-03-24T11:20:17.718394+0000","last_fullsized":"2024-03-24T11:20:17.718394+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070486,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32516,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712636,"statfs":{"total":95995035648,"available":95961739264,"internally_reserved":0,"allocated":1372160,"data_stored":963547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364827,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1359872,"data_stored":963768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":806912,"data_stored":374237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1380352,"data_stored":963302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":774144,"data_stored":372695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":373429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":790528,"data_stored":373607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31928,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713224,"statfs":{"total":95995035648,"available":95962341376,"internally_reserved":0,"allocated":770048,"data_stored":372961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:20:26.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:20:26.634 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:27.017 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:26 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:20:27.017 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:26 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:20:27.017 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:26 smithi181 ceph-mon[33998]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:27.065 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:26 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:20:27.065 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:26 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:20:27.065 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:26 smithi179 ceph-mon[29608]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:27.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:26 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:20:27.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:26 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:20:27.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:26 smithi179 ceph-mon[37907]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 6.b scrub starts 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 6.b scrub ok 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:20:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 6.b scrub starts 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 6.b scrub ok 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:20:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:27 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 6.b scrub starts 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 6.b scrub ok 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:20:28.056 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:27 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:20:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:20:28.524 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:20:28.766 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:20:28.767 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:20:28.767 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 3.1c deep-scrub starts 2024-03-24T11:20:28.767 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 3.1c deep-scrub ok 2024-03-24T11:20:28.767 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:20:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:28 smithi181 podman[64792]: 2024-03-24 11:20:28.667498705 +0000 UTC m=+0.120542450 container create 348827ab325965fa7dcd6bd382b60c0b34934551f0a9843d857d9a27989af10f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.buildah.version=1.32.0, name=ubi8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8) 2024-03-24T11:20:29.058 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:28 smithi181 podman[64792]: 2024-03-24 11:20:28.895472124 +0000 UTC m=+0.348515835 container init 348827ab325965fa7dcd6bd382b60c0b34934551f0a9843d857d9a27989af10f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.openshift.tags=base rhel8, name=ubi8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-24T11:20:29.058 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:28 smithi181 podman[64792]: 2024-03-24 11:20:28.898231226 +0000 UTC m=+0.351274926 container start 348827ab325965fa7dcd6bd382b60c0b34934551f0a9843d857d9a27989af10f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, description=Ceph Grafana Container, vendor=Red Hat, Inc., distribution-scope=public, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, name=ubi8) 2024-03-24T11:20:29.058 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:28 smithi181 bash[64792]: 348827ab325965fa7dcd6bd382b60c0b34934551f0a9843d857d9a27989af10f 2024-03-24T11:20:29.059 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:28 smithi181 systemd[1]: Started Ceph grafana.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 3.1c deep-scrub starts 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 3.1c deep-scrub ok 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:20:29.059 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:28 smithi181 ceph-mon[33998]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:29.118 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:20:29.118 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[29608]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 3.1c deep-scrub starts 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 3.1c deep-scrub ok 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 996 B/s rd, 0 op/s 2024-03-24T11:20:29.119 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:20:29.120 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:20:29.120 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:28 smithi179 ceph-mon[37907]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:20:29.148 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-24T11:20:27.989926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1016,"num_read_kb":862,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":613,"ondisk_log_size":613,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249204,"kb_used_data":7868,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749712012,"statfs":{"total":767960285184,"available":767705100288,"internally_reserved":0,"allocated":8056832,"data_stored":4757546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.273579"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717940+0000","last_change":"2024-03-24T11:18:59.704400+0000","last_active":"2024-03-24T11:20:17.717940+0000","last_peered":"2024-03-24T11:20:17.717940+0000","last_clean":"2024-03-24T11:20:17.717940+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:20:17.717940+0000","last_undegraded":"2024-03-24T11:20:17.717940+0000","last_fullsized":"2024-03-24T11:20:17.717940+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:22.357471+0000","last_change":"2024-03-24T11:20:22.357494+0000","last_active":"2024-03-24T11:20:22.357471+0000","last_peered":"2024-03-24T11:20:22.357471+0000","last_clean":"2024-03-24T11:20:22.357471+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:20:22.357471+0000","last_undegraded":"2024-03-24T11:20:22.357471+0000","last_fullsized":"2024-03-24T11:20:22.357471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:22.357393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:22.357393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:44.024523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721003+0000","last_change":"2024-03-24T11:18:53.684875+0000","last_active":"2024-03-24T11:20:17.721003+0000","last_peered":"2024-03-24T11:20:17.721003+0000","last_clean":"2024-03-24T11:20:17.721003+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:20:17.721003+0000","last_undegraded":"2024-03-24T11:20:17.721003+0000","last_fullsized":"2024-03-24T11:20:17.721003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:18.084479+0000","last_change":"2024-03-24T11:20:18.084538+0000","last_active":"2024-03-24T11:20:18.084479+0000","last_peered":"2024-03-24T11:20:18.084479+0000","last_clean":"2024-03-24T11:20:18.084479+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:20:18.084479+0000","last_undegraded":"2024-03-24T11:20:18.084479+0000","last_fullsized":"2024-03-24T11:20:18.084479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:20:18.084379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:18.084379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:40.748807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727934+0000","last_change":"2024-03-24T11:19:00.314493+0000","last_active":"2024-03-24T11:20:17.727934+0000","last_peered":"2024-03-24T11:20:17.727934+0000","last_clean":"2024-03-24T11:20:17.727934+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:20:17.727934+0000","last_undegraded":"2024-03-24T11:20:17.727934+0000","last_fullsized":"2024-03-24T11:20:17.727934+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.788301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717485+0000","last_change":"2024-03-24T11:18:57.698383+0000","last_active":"2024-03-24T11:20:17.717485+0000","last_peered":"2024-03-24T11:20:17.717485+0000","last_clean":"2024-03-24T11:20:17.717485+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:20:17.717485+0000","last_undegraded":"2024-03-24T11:20:17.717485+0000","last_fullsized":"2024-03-24T11:20:17.717485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728525+0000","last_change":"2024-03-24T11:18:53.680266+0000","last_active":"2024-03-24T11:20:17.728525+0000","last_peered":"2024-03-24T11:20:17.728525+0000","last_clean":"2024-03-24T11:20:17.728525+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:20:17.728525+0000","last_undegraded":"2024-03-24T11:20:17.728525+0000","last_fullsized":"2024-03-24T11:20:17.728525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727754+0000","last_change":"2024-03-24T11:18:55.680984+0000","last_active":"2024-03-24T11:20:17.727754+0000","last_peered":"2024-03-24T11:20:17.727754+0000","last_clean":"2024-03-24T11:20:17.727754+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:20:17.727754+0000","last_undegraded":"2024-03-24T11:20:17.727754+0000","last_fullsized":"2024-03-24T11:20:17.727754+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:24.173013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.068322+0000","last_change":"2024-03-24T11:20:20.068342+0000","last_active":"2024-03-24T11:20:20.068322+0000","last_peered":"2024-03-24T11:20:20.068322+0000","last_clean":"2024-03-24T11:20:20.068322+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:20:20.068322+0000","last_undegraded":"2024-03-24T11:20:20.068322+0000","last_fullsized":"2024-03-24T11:20:20.068322+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:20.068266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.068266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:35.574643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724680+0000","last_change":"2024-03-24T11:20:05.382085+0000","last_active":"2024-03-24T11:20:17.724680+0000","last_peered":"2024-03-24T11:20:17.724680+0000","last_clean":"2024-03-24T11:20:17.724680+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:20:17.724680+0000","last_undegraded":"2024-03-24T11:20:17.724680+0000","last_fullsized":"2024-03-24T11:20:17.724680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.381936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.381936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.568574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:18.374322+0000","last_change":"2024-03-24T11:20:18.374342+0000","last_active":"2024-03-24T11:20:18.374322+0000","last_peered":"2024-03-24T11:20:18.374322+0000","last_clean":"2024-03-24T11:20:18.374322+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:20:18.374322+0000","last_undegraded":"2024-03-24T11:20:18.374322+0000","last_fullsized":"2024-03-24T11:20:18.374322+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:18.374262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:18.374262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:43.607532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718059+0000","last_change":"2024-03-24T11:18:55.682437+0000","last_active":"2024-03-24T11:20:17.718059+0000","last_peered":"2024-03-24T11:20:17.718059+0000","last_clean":"2024-03-24T11:20:17.718059+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:20:17.718059+0000","last_undegraded":"2024-03-24T11:20:17.718059+0000","last_fullsized":"2024-03-24T11:20:17.718059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.808265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724501+0000","last_change":"2024-03-24T11:18:59.707100+0000","last_active":"2024-03-24T11:20:17.724501+0000","last_peered":"2024-03-24T11:20:17.724501+0000","last_clean":"2024-03-24T11:20:17.724501+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:20:17.724501+0000","last_undegraded":"2024-03-24T11:20:17.724501+0000","last_fullsized":"2024-03-24T11:20:17.724501+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725164+0000","last_change":"2024-03-24T11:18:57.701380+0000","last_active":"2024-03-24T11:20:17.725164+0000","last_peered":"2024-03-24T11:20:17.725164+0000","last_clean":"2024-03-24T11:20:17.725164+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:20:17.725164+0000","last_undegraded":"2024-03-24T11:20:17.725164+0000","last_fullsized":"2024-03-24T11:20:17.725164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724758+0000","last_change":"2024-03-24T11:18:53.681681+0000","last_active":"2024-03-24T11:20:17.724758+0000","last_peered":"2024-03-24T11:20:17.724758+0000","last_clean":"2024-03-24T11:20:17.724758+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:20:17.724758+0000","last_undegraded":"2024-03-24T11:20:17.724758+0000","last_fullsized":"2024-03-24T11:20:17.724758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718005+0000","last_change":"2024-03-24T11:18:55.686558+0000","last_active":"2024-03-24T11:20:17.718005+0000","last_peered":"2024-03-24T11:20:17.718005+0000","last_clean":"2024-03-24T11:20:17.718005+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:20:17.718005+0000","last_undegraded":"2024-03-24T11:20:17.718005+0000","last_fullsized":"2024-03-24T11:20:17.718005+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.670081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718719+0000","last_change":"2024-03-24T11:20:00.997176+0000","last_active":"2024-03-24T11:20:17.718719+0000","last_peered":"2024-03-24T11:20:17.718719+0000","last_clean":"2024-03-24T11:20:17.718719+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:20:17.718719+0000","last_undegraded":"2024-03-24T11:20:17.718719+0000","last_fullsized":"2024-03-24T11:20:17.718719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.997096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.997096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.143056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720937+0000","last_change":"2024-03-24T11:20:15.352309+0000","last_active":"2024-03-24T11:20:17.720937+0000","last_peered":"2024-03-24T11:20:17.720937+0000","last_clean":"2024-03-24T11:20:17.720937+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:20:17.720937+0000","last_undegraded":"2024-03-24T11:20:17.720937+0000","last_fullsized":"2024-03-24T11:20:17.720937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:15.352224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:15.352224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:36.853195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721464+0000","last_change":"2024-03-24T11:18:53.681845+0000","last_active":"2024-03-24T11:20:17.721464+0000","last_peered":"2024-03-24T11:20:17.721464+0000","last_clean":"2024-03-24T11:20:17.721464+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:20:17.721464+0000","last_undegraded":"2024-03-24T11:20:17.721464+0000","last_fullsized":"2024-03-24T11:20:17.721464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724932+0000","last_change":"2024-03-24T11:20:11.516126+0000","last_active":"2024-03-24T11:20:17.724932+0000","last_peered":"2024-03-24T11:20:17.724932+0000","last_clean":"2024-03-24T11:20:17.724932+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:20:17.724932+0000","last_undegraded":"2024-03-24T11:20:17.724932+0000","last_fullsized":"2024-03-24T11:20:17.724932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:20:11.516026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:11.516026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.641243+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728352+0000","last_change":"2024-03-24T11:19:00.312267+0000","last_active":"2024-03-24T11:20:17.728352+0000","last_peered":"2024-03-24T11:20:17.728352+0000","last_clean":"2024-03-24T11:20:17.728352+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:20:17.728352+0000","last_undegraded":"2024-03-24T11:20:17.728352+0000","last_fullsized":"2024-03-24T11:20:17.728352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:17.807373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718115+0000","last_change":"2024-03-24T11:18:57.699401+0000","last_active":"2024-03-24T11:20:17.718115+0000","last_peered":"2024-03-24T11:20:17.718115+0000","last_clean":"2024-03-24T11:20:17.718115+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:20:17.718115+0000","last_undegraded":"2024-03-24T11:20:17.718115+0000","last_fullsized":"2024-03-24T11:20:17.718115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718081+0000","last_change":"2024-03-24T11:18:53.672450+0000","last_active":"2024-03-24T11:20:17.718081+0000","last_peered":"2024-03-24T11:20:17.718081+0000","last_clean":"2024-03-24T11:20:17.718081+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:20:17.718081+0000","last_undegraded":"2024-03-24T11:20:17.718081+0000","last_fullsized":"2024-03-24T11:20:17.718081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728180+0000","last_change":"2024-03-24T11:18:55.684714+0000","last_active":"2024-03-24T11:20:17.728180+0000","last_peered":"2024-03-24T11:20:17.728180+0000","last_clean":"2024-03-24T11:20:17.728180+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:20:17.728180+0000","last_undegraded":"2024-03-24T11:20:17.728180+0000","last_fullsized":"2024-03-24T11:20:17.728180+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.614052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:22.871913+0000","last_change":"2024-03-24T11:20:22.871965+0000","last_active":"2024-03-24T11:20:22.871913+0000","last_peered":"2024-03-24T11:20:22.871913+0000","last_clean":"2024-03-24T11:20:22.871913+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:20:22.871913+0000","last_undegraded":"2024-03-24T11:20:22.871913+0000","last_fullsized":"2024-03-24T11:20:22.871913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:22.871822+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:45.749060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727563+0000","last_change":"2024-03-24T11:20:11.338061+0000","last_active":"2024-03-24T11:20:17.727563+0000","last_peered":"2024-03-24T11:20:17.727563+0000","last_clean":"2024-03-24T11:20:17.727563+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:20:17.727563+0000","last_undegraded":"2024-03-24T11:20:17.727563+0000","last_fullsized":"2024-03-24T11:20:17.727563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:20:11.337962+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:20:11.337962+0000","last_clean_scrub_stamp":"2024-03-24T11:20:11.337962+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:34.099110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722239+0000","last_change":"2024-03-24T11:20:01.218213+0000","last_active":"2024-03-24T11:20:17.722239+0000","last_peered":"2024-03-24T11:20:17.722239+0000","last_clean":"2024-03-24T11:20:17.722239+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:20:17.722239+0000","last_undegraded":"2024-03-24T11:20:17.722239+0000","last_fullsized":"2024-03-24T11:20:17.722239+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:01.218061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.218061+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.318777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721941+0000","last_change":"2024-03-24T11:20:03.001173+0000","last_active":"2024-03-24T11:20:17.721941+0000","last_peered":"2024-03-24T11:20:17.721941+0000","last_clean":"2024-03-24T11:20:17.721941+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:20:17.721941+0000","last_undegraded":"2024-03-24T11:20:17.721941+0000","last_fullsized":"2024-03-24T11:20:17.721941+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:20:03.001079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.001079+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.140666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721084+0000","last_change":"2024-03-24T11:20:05.208928+0000","last_active":"2024-03-24T11:20:17.721084+0000","last_peered":"2024-03-24T11:20:17.721084+0000","last_clean":"2024-03-24T11:20:17.721084+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:20:17.721084+0000","last_undegraded":"2024-03-24T11:20:17.721084+0000","last_fullsized":"2024-03-24T11:20:17.721084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.208832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.208832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:28.229456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722066+0000","last_change":"2024-03-24T11:20:10.934360+0000","last_active":"2024-03-24T11:20:17.722066+0000","last_peered":"2024-03-24T11:20:17.722066+0000","last_clean":"2024-03-24T11:20:17.722066+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:20:17.722066+0000","last_undegraded":"2024-03-24T11:20:17.722066+0000","last_fullsized":"2024-03-24T11:20:17.722066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:10.934216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:10.934216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:33.120547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727493+0000","last_change":"2024-03-24T11:19:57.458406+0000","last_active":"2024-03-24T11:20:17.727493+0000","last_peered":"2024-03-24T11:20:17.727493+0000","last_clean":"2024-03-24T11:20:17.727493+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:20:17.727493+0000","last_undegraded":"2024-03-24T11:20:17.727493+0000","last_fullsized":"2024-03-24T11:20:17.727493+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:57.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:57.458291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.184542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":83,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727447+0000","last_change":"2024-03-24T11:20:12.347839+0000","last_active":"2024-03-24T11:20:17.727447+0000","last_peered":"2024-03-24T11:20:17.727447+0000","last_clean":"2024-03-24T11:20:17.727447+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:20:17.727447+0000","last_undegraded":"2024-03-24T11:20:17.727447+0000","last_fullsized":"2024-03-24T11:20:17.727447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:20:12.347712+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:20:12.347712+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.347712+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.610126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725064+0000","last_change":"2024-03-24T11:20:03.896346+0000","last_active":"2024-03-24T11:20:17.725064+0000","last_peered":"2024-03-24T11:20:17.725064+0000","last_clean":"2024-03-24T11:20:17.725064+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:20:17.725064+0000","last_undegraded":"2024-03-24T11:20:17.725064+0000","last_fullsized":"2024-03-24T11:20:17.725064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.896225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.896225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.959693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725295+0000","last_change":"2024-03-24T11:18:57.700812+0000","last_active":"2024-03-24T11:20:17.725295+0000","last_peered":"2024-03-24T11:20:17.725295+0000","last_clean":"2024-03-24T11:20:17.725295+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:20:17.725295+0000","last_undegraded":"2024-03-24T11:20:17.725295+0000","last_fullsized":"2024-03-24T11:20:17.725295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:25.753316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720979+0000","last_change":"2024-03-24T11:20:07.018200+0000","last_active":"2024-03-24T11:20:17.720979+0000","last_peered":"2024-03-24T11:20:17.720979+0000","last_clean":"2024-03-24T11:20:17.720979+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:20:17.720979+0000","last_undegraded":"2024-03-24T11:20:17.720979+0000","last_fullsized":"2024-03-24T11:20:17.720979+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.018119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.018119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.136141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"87'6","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727499+0000","last_change":"2024-03-24T11:20:14.327248+0000","last_active":"2024-03-24T11:20:17.727499+0000","last_peered":"2024-03-24T11:20:17.727499+0000","last_clean":"2024-03-24T11:20:17.727499+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:20:17.727499+0000","last_undegraded":"2024-03-24T11:20:17.727499+0000","last_fullsized":"2024-03-24T11:20:17.727499+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'6","last_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:20:14.327154+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:35.752212+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727829+0000","last_change":"2024-03-24T11:19:00.314378+0000","last_active":"2024-03-24T11:20:17.727829+0000","last_peered":"2024-03-24T11:20:17.727829+0000","last_clean":"2024-03-24T11:20:17.727829+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:20:17.727829+0000","last_undegraded":"2024-03-24T11:20:17.727829+0000","last_fullsized":"2024-03-24T11:20:17.727829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727165+0000","last_change":"2024-03-24T11:18:57.698486+0000","last_active":"2024-03-24T11:20:17.727165+0000","last_peered":"2024-03-24T11:20:17.727165+0000","last_clean":"2024-03-24T11:20:17.727165+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:20:17.727165+0000","last_undegraded":"2024-03-24T11:20:17.727165+0000","last_fullsized":"2024-03-24T11:20:17.727165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725848+0000","last_change":"2024-03-24T11:20:16.429420+0000","last_active":"2024-03-24T11:20:17.725848+0000","last_peered":"2024-03-24T11:20:17.725848+0000","last_clean":"2024-03-24T11:20:17.725848+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:20:17.725848+0000","last_undegraded":"2024-03-24T11:20:17.725848+0000","last_fullsized":"2024-03-24T11:20:17.725848+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:16.429342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:16.429342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.285148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727699+0000","last_change":"2024-03-24T11:18:56.141810+0000","last_active":"2024-03-24T11:20:17.727699+0000","last_peered":"2024-03-24T11:20:17.727699+0000","last_clean":"2024-03-24T11:20:17.727699+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:20:17.727699+0000","last_undegraded":"2024-03-24T11:20:17.727699+0000","last_fullsized":"2024-03-24T11:20:17.727699+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720134+0000","last_change":"2024-03-24T11:18:59.714114+0000","last_active":"2024-03-24T11:20:17.720134+0000","last_peered":"2024-03-24T11:20:17.720134+0000","last_clean":"2024-03-24T11:20:17.720134+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:20:17.720134+0000","last_undegraded":"2024-03-24T11:20:17.720134+0000","last_fullsized":"2024-03-24T11:20:17.720134+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.411365+0000","last_change":"2024-03-24T11:20:20.411389+0000","last_active":"2024-03-24T11:20:20.411365+0000","last_peered":"2024-03-24T11:20:20.411365+0000","last_clean":"2024-03-24T11:20:20.411365+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:20:20.411365+0000","last_undegraded":"2024-03-24T11:20:20.411365+0000","last_fullsized":"2024-03-24T11:20:20.411365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:20.411287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.411287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.233362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726215+0000","last_change":"2024-03-24T11:18:53.681700+0000","last_active":"2024-03-24T11:20:17.726215+0000","last_peered":"2024-03-24T11:20:17.726215+0000","last_clean":"2024-03-24T11:20:17.726215+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:20:17.726215+0000","last_undegraded":"2024-03-24T11:20:17.726215+0000","last_fullsized":"2024-03-24T11:20:17.726215+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.896550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722390+0000","last_change":"2024-03-24T11:18:55.683808+0000","last_active":"2024-03-24T11:20:17.722390+0000","last_peered":"2024-03-24T11:20:17.722390+0000","last_clean":"2024-03-24T11:20:17.722390+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:20:17.722390+0000","last_undegraded":"2024-03-24T11:20:17.722390+0000","last_fullsized":"2024-03-24T11:20:17.722390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727515+0000","last_change":"2024-03-24T11:20:03.441868+0000","last_active":"2024-03-24T11:20:17.727515+0000","last_peered":"2024-03-24T11:20:17.727515+0000","last_clean":"2024-03-24T11:20:17.727515+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:20:17.727515+0000","last_undegraded":"2024-03-24T11:20:17.727515+0000","last_fullsized":"2024-03-24T11:20:17.727515+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:03.441791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.441791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.538293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720795+0000","last_change":"2024-03-24T11:18:57.703012+0000","last_active":"2024-03-24T11:20:17.720795+0000","last_peered":"2024-03-24T11:20:17.720795+0000","last_clean":"2024-03-24T11:20:17.720795+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:20:17.720795+0000","last_undegraded":"2024-03-24T11:20:17.720795+0000","last_fullsized":"2024-03-24T11:20:17.720795+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.320906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718306+0000","last_change":"2024-03-24T11:19:56.100757+0000","last_active":"2024-03-24T11:20:17.718306+0000","last_peered":"2024-03-24T11:20:17.718306+0000","last_clean":"2024-03-24T11:20:17.718306+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:20:17.718306+0000","last_undegraded":"2024-03-24T11:20:17.718306+0000","last_fullsized":"2024-03-24T11:20:17.718306+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.100633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.072386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718274+0000","last_change":"2024-03-24T11:20:04.036645+0000","last_active":"2024-03-24T11:20:17.718274+0000","last_peered":"2024-03-24T11:20:17.718274+0000","last_clean":"2024-03-24T11:20:17.718274+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:20:17.718274+0000","last_undegraded":"2024-03-24T11:20:17.718274+0000","last_fullsized":"2024-03-24T11:20:17.718274+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:20:04.036510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.036510+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.183695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721407+0000","last_change":"2024-03-24T11:18:59.717539+0000","last_active":"2024-03-24T11:20:17.721407+0000","last_peered":"2024-03-24T11:20:17.721407+0000","last_clean":"2024-03-24T11:20:17.721407+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:20:17.721407+0000","last_undegraded":"2024-03-24T11:20:17.721407+0000","last_fullsized":"2024-03-24T11:20:17.721407+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.956051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725276+0000","last_change":"2024-03-24T11:20:07.390710+0000","last_active":"2024-03-24T11:20:17.725276+0000","last_peered":"2024-03-24T11:20:17.725276+0000","last_clean":"2024-03-24T11:20:17.725276+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:20:17.725276+0000","last_undegraded":"2024-03-24T11:20:17.725276+0000","last_fullsized":"2024-03-24T11:20:17.725276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:07.390604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.390604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.892391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725427+0000","last_change":"2024-03-24T11:18:53.682165+0000","last_active":"2024-03-24T11:20:17.725427+0000","last_peered":"2024-03-24T11:20:17.725427+0000","last_clean":"2024-03-24T11:20:17.725427+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:20:17.725427+0000","last_undegraded":"2024-03-24T11:20:17.725427+0000","last_fullsized":"2024-03-24T11:20:17.725427+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.317744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728228+0000","last_change":"2024-03-24T11:18:56.141308+0000","last_active":"2024-03-24T11:20:17.728228+0000","last_peered":"2024-03-24T11:20:17.728228+0000","last_clean":"2024-03-24T11:20:17.728228+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:20:17.728228+0000","last_undegraded":"2024-03-24T11:20:17.728228+0000","last_fullsized":"2024-03-24T11:20:17.728228+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718789+0000","last_change":"2024-03-24T11:18:59.705914+0000","last_active":"2024-03-24T11:20:17.718789+0000","last_peered":"2024-03-24T11:20:17.718789+0000","last_clean":"2024-03-24T11:20:17.718789+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:20:17.718789+0000","last_undegraded":"2024-03-24T11:20:17.718789+0000","last_fullsized":"2024-03-24T11:20:17.718789+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728416+0000","last_change":"2024-03-24T11:20:04.436584+0000","last_active":"2024-03-24T11:20:17.728416+0000","last_peered":"2024-03-24T11:20:17.728416+0000","last_clean":"2024-03-24T11:20:17.728416+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:20:17.728416+0000","last_undegraded":"2024-03-24T11:20:17.728416+0000","last_fullsized":"2024-03-24T11:20:17.728416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:04.436493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.436493+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.892034+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721104+0000","last_change":"2024-03-24T11:18:53.683017+0000","last_active":"2024-03-24T11:20:17.721104+0000","last_peered":"2024-03-24T11:20:17.721104+0000","last_clean":"2024-03-24T11:20:17.721104+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:20:17.721104+0000","last_undegraded":"2024-03-24T11:20:17.721104+0000","last_fullsized":"2024-03-24T11:20:17.721104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725463+0000","last_change":"2024-03-24T11:18:56.143374+0000","last_active":"2024-03-24T11:20:17.725463+0000","last_peered":"2024-03-24T11:20:17.725463+0000","last_clean":"2024-03-24T11:20:17.725463+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:20:17.725463+0000","last_undegraded":"2024-03-24T11:20:17.725463+0000","last_fullsized":"2024-03-24T11:20:17.725463+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:21.462286+0000","last_change":"2024-03-24T11:20:21.462307+0000","last_active":"2024-03-24T11:20:21.462286+0000","last_peered":"2024-03-24T11:20:21.462286+0000","last_clean":"2024-03-24T11:20:21.462286+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:20:21.462286+0000","last_undegraded":"2024-03-24T11:20:21.462286+0000","last_fullsized":"2024-03-24T11:20:21.462286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:21.462219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:21.462219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.223620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728652+0000","last_change":"2024-03-24T11:18:57.694184+0000","last_active":"2024-03-24T11:20:17.728652+0000","last_peered":"2024-03-24T11:20:17.728652+0000","last_clean":"2024-03-24T11:20:17.728652+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:20:17.728652+0000","last_undegraded":"2024-03-24T11:20:17.728652+0000","last_fullsized":"2024-03-24T11:20:17.728652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725168+0000","last_change":"2024-03-24T11:19:58.799434+0000","last_active":"2024-03-24T11:20:17.725168+0000","last_peered":"2024-03-24T11:20:17.725168+0000","last_clean":"2024-03-24T11:20:17.725168+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:20:17.725168+0000","last_undegraded":"2024-03-24T11:20:17.725168+0000","last_fullsized":"2024-03-24T11:20:17.725168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.799295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.799295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.674050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721944+0000","last_change":"2024-03-24T11:19:58.001386+0000","last_active":"2024-03-24T11:20:17.721944+0000","last_peered":"2024-03-24T11:20:17.721944+0000","last_clean":"2024-03-24T11:20:17.721944+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:20:17.721944+0000","last_undegraded":"2024-03-24T11:20:17.721944+0000","last_fullsized":"2024-03-24T11:20:17.721944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.001296+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.932223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726170+0000","last_change":"2024-03-24T11:19:00.314565+0000","last_active":"2024-03-24T11:20:17.726170+0000","last_peered":"2024-03-24T11:20:17.726170+0000","last_clean":"2024-03-24T11:20:17.726170+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:20:17.726170+0000","last_undegraded":"2024-03-24T11:20:17.726170+0000","last_fullsized":"2024-03-24T11:20:17.726170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.091963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725927+0000","last_change":"2024-03-24T11:18:57.693993+0000","last_active":"2024-03-24T11:20:17.725927+0000","last_peered":"2024-03-24T11:20:17.725927+0000","last_clean":"2024-03-24T11:20:17.725927+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:20:17.725927+0000","last_undegraded":"2024-03-24T11:20:17.725927+0000","last_fullsized":"2024-03-24T11:20:17.725927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.747158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725214+0000","last_change":"2024-03-24T11:20:01.231696+0000","last_active":"2024-03-24T11:20:17.725214+0000","last_peered":"2024-03-24T11:20:17.725214+0000","last_clean":"2024-03-24T11:20:17.725214+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:20:17.725214+0000","last_undegraded":"2024-03-24T11:20:17.725214+0000","last_fullsized":"2024-03-24T11:20:17.725214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:01.231617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.231617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.010965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":60,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728656+0000","last_change":"2024-03-24T11:18:56.142569+0000","last_active":"2024-03-24T11:20:17.728656+0000","last_peered":"2024-03-24T11:20:17.728656+0000","last_clean":"2024-03-24T11:20:17.728656+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:20:17.728656+0000","last_undegraded":"2024-03-24T11:20:17.728656+0000","last_fullsized":"2024-03-24T11:20:17.728656+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.660826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724852+0000","last_change":"2024-03-24T11:20:02.408219+0000","last_active":"2024-03-24T11:20:17.724852+0000","last_peered":"2024-03-24T11:20:17.724852+0000","last_clean":"2024-03-24T11:20:17.724852+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:20:17.724852+0000","last_undegraded":"2024-03-24T11:20:17.724852+0000","last_fullsized":"2024-03-24T11:20:17.724852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.408130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.408130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:03.752282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726601+0000","last_change":"2024-03-24T11:20:15.827098+0000","last_active":"2024-03-24T11:20:17.726601+0000","last_peered":"2024-03-24T11:20:17.726601+0000","last_clean":"2024-03-24T11:20:17.726601+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:20:17.726601+0000","last_undegraded":"2024-03-24T11:20:17.726601+0000","last_fullsized":"2024-03-24T11:20:17.726601+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-24T11:20:15.827014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:15.827014+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:26.060294+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724471+0000","last_change":"2024-03-24T11:20:12.354666+0000","last_active":"2024-03-24T11:20:17.724471+0000","last_peered":"2024-03-24T11:20:17.724471+0000","last_clean":"2024-03-24T11:20:17.724471+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:20:17.724471+0000","last_undegraded":"2024-03-24T11:20:17.724471+0000","last_fullsized":"2024-03-24T11:20:17.724471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:20:12.354559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.354559+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.930978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725443+0000","last_change":"2024-03-24T11:20:05.863734+0000","last_active":"2024-03-24T11:20:17.725443+0000","last_peered":"2024-03-24T11:20:17.725443+0000","last_clean":"2024-03-24T11:20:17.725443+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:20:17.725443+0000","last_undegraded":"2024-03-24T11:20:17.725443+0000","last_fullsized":"2024-03-24T11:20:17.725443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:05.863616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.863616+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:09.078050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727862+0000","last_change":"2024-03-24T11:18:59.706989+0000","last_active":"2024-03-24T11:20:17.727862+0000","last_peered":"2024-03-24T11:20:17.727862+0000","last_clean":"2024-03-24T11:20:17.727862+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:20:17.727862+0000","last_undegraded":"2024-03-24T11:20:17.727862+0000","last_fullsized":"2024-03-24T11:20:17.727862+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726606+0000","last_change":"2024-03-24T11:18:53.682847+0000","last_active":"2024-03-24T11:20:17.726606+0000","last_peered":"2024-03-24T11:20:17.726606+0000","last_clean":"2024-03-24T11:20:17.726606+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:20:17.726606+0000","last_undegraded":"2024-03-24T11:20:17.726606+0000","last_fullsized":"2024-03-24T11:20:17.726606+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.562909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.132832+0000","last_change":"2024-03-24T11:20:20.132861+0000","last_active":"2024-03-24T11:20:20.132832+0000","last_peered":"2024-03-24T11:20:20.132832+0000","last_clean":"2024-03-24T11:20:20.132832+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:20:20.132832+0000","last_undegraded":"2024-03-24T11:20:20.132832+0000","last_fullsized":"2024-03-24T11:20:20.132832+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:20:20.132772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.132772+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:40.124937+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724647+0000","last_change":"2024-03-24T11:18:57.704315+0000","last_active":"2024-03-24T11:20:17.724647+0000","last_peered":"2024-03-24T11:20:17.724647+0000","last_clean":"2024-03-24T11:20:17.724647+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:20:17.724647+0000","last_undegraded":"2024-03-24T11:20:17.724647+0000","last_fullsized":"2024-03-24T11:20:17.724647+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.986639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726889+0000","last_change":"2024-03-24T11:18:59.707271+0000","last_active":"2024-03-24T11:20:17.726889+0000","last_peered":"2024-03-24T11:20:17.726889+0000","last_clean":"2024-03-24T11:20:17.726889+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:20:17.726889+0000","last_undegraded":"2024-03-24T11:20:17.726889+0000","last_fullsized":"2024-03-24T11:20:17.726889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.298674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:20.474333+0000","last_change":"2024-03-24T11:20:20.474352+0000","last_active":"2024-03-24T11:20:20.474333+0000","last_peered":"2024-03-24T11:20:20.474333+0000","last_clean":"2024-03-24T11:20:20.474333+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:20:20.474333+0000","last_undegraded":"2024-03-24T11:20:20.474333+0000","last_fullsized":"2024-03-24T11:20:20.474333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:20.474279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:20.474279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:33.479751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726315+0000","last_change":"2024-03-24T11:20:00.851820+0000","last_active":"2024-03-24T11:20:17.726315+0000","last_peered":"2024-03-24T11:20:17.726315+0000","last_clean":"2024-03-24T11:20:17.726315+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:20:17.726315+0000","last_undegraded":"2024-03-24T11:20:17.726315+0000","last_fullsized":"2024-03-24T11:20:17.726315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.851691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.851691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:07.342041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726280+0000","last_change":"2024-03-24T11:18:56.143415+0000","last_active":"2024-03-24T11:20:17.726280+0000","last_peered":"2024-03-24T11:20:17.726280+0000","last_clean":"2024-03-24T11:20:17.726280+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:20:17.726280+0000","last_undegraded":"2024-03-24T11:20:17.726280+0000","last_fullsized":"2024-03-24T11:20:17.726280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:18.105644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720393+0000","last_change":"2024-03-24T11:20:07.980193+0000","last_active":"2024-03-24T11:20:17.720393+0000","last_peered":"2024-03-24T11:20:17.720393+0000","last_clean":"2024-03-24T11:20:17.720393+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:20:17.720393+0000","last_undegraded":"2024-03-24T11:20:17.720393+0000","last_fullsized":"2024-03-24T11:20:17.720393+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.980113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.980113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.058894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724832+0000","last_change":"2024-03-24T11:20:13.558725+0000","last_active":"2024-03-24T11:20:17.724832+0000","last_peered":"2024-03-24T11:20:17.724832+0000","last_clean":"2024-03-24T11:20:17.724832+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:20:17.724832+0000","last_undegraded":"2024-03-24T11:20:17.724832+0000","last_fullsized":"2024-03-24T11:20:17.724832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:13.558645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:13.558645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:23.034282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724808+0000","last_change":"2024-03-24T11:19:56.404250+0000","last_active":"2024-03-24T11:20:17.724808+0000","last_peered":"2024-03-24T11:20:17.724808+0000","last_clean":"2024-03-24T11:20:17.724808+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:20:17.724808+0000","last_undegraded":"2024-03-24T11:20:17.724808+0000","last_fullsized":"2024-03-24T11:20:17.724808+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:19:56.404146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:56.404146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.193173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721637+0000","last_change":"2024-03-24T11:18:55.684166+0000","last_active":"2024-03-24T11:20:17.721637+0000","last_peered":"2024-03-24T11:20:17.721637+0000","last_clean":"2024-03-24T11:20:17.721637+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:20:17.721637+0000","last_undegraded":"2024-03-24T11:20:17.721637+0000","last_fullsized":"2024-03-24T11:20:17.721637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:19.368389+0000","last_change":"2024-03-24T11:20:19.368409+0000","last_active":"2024-03-24T11:20:19.368389+0000","last_peered":"2024-03-24T11:20:19.368389+0000","last_clean":"2024-03-24T11:20:19.368389+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:20:19.368389+0000","last_undegraded":"2024-03-24T11:20:19.368389+0000","last_fullsized":"2024-03-24T11:20:19.368389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:19.368309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:19.368309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:42.640854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724564+0000","last_change":"2024-03-24T11:18:56.680518+0000","last_active":"2024-03-24T11:20:17.724564+0000","last_peered":"2024-03-24T11:20:17.724564+0000","last_clean":"2024-03-24T11:20:17.724564+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:20:17.724564+0000","last_undegraded":"2024-03-24T11:20:17.724564+0000","last_fullsized":"2024-03-24T11:20:17.724564+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:18:51.653668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718111+0000","last_change":"2024-03-24T11:18:53.680619+0000","last_active":"2024-03-24T11:20:17.718111+0000","last_peered":"2024-03-24T11:20:17.718111+0000","last_clean":"2024-03-24T11:20:17.718111+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:20:17.718111+0000","last_undegraded":"2024-03-24T11:20:17.718111+0000","last_fullsized":"2024-03-24T11:20:17.718111+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":125,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722368+0000","last_change":"2024-03-24T11:19:55.086981+0000","last_active":"2024-03-24T11:20:17.722368+0000","last_peered":"2024-03-24T11:20:17.722368+0000","last_clean":"2024-03-24T11:20:17.722368+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:20:17.722368+0000","last_undegraded":"2024-03-24T11:20:17.722368+0000","last_fullsized":"2024-03-24T11:20:17.722368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:19:55.086847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:19:55.086847+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:14.068671+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720596+0000","last_change":"2024-03-24T11:18:57.694386+0000","last_active":"2024-03-24T11:20:17.720596+0000","last_peered":"2024-03-24T11:20:17.720596+0000","last_clean":"2024-03-24T11:20:17.720596+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:20:17.720596+0000","last_undegraded":"2024-03-24T11:20:17.720596+0000","last_fullsized":"2024-03-24T11:20:17.720596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726397+0000","last_change":"2024-03-24T11:20:12.833645+0000","last_active":"2024-03-24T11:20:17.726397+0000","last_peered":"2024-03-24T11:20:17.726397+0000","last_clean":"2024-03-24T11:20:17.726397+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:20:17.726397+0000","last_undegraded":"2024-03-24T11:20:17.726397+0000","last_fullsized":"2024-03-24T11:20:17.726397+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:12.833509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:12.833509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.607503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721577+0000","last_change":"2024-03-24T11:19:58.185937+0000","last_active":"2024-03-24T11:20:17.721577+0000","last_peered":"2024-03-24T11:20:17.721577+0000","last_clean":"2024-03-24T11:20:17.721577+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:20:17.721577+0000","last_undegraded":"2024-03-24T11:20:17.721577+0000","last_fullsized":"2024-03-24T11:20:17.721577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:19:58.185832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:19:58.185832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:02.274032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724784+0000","last_change":"2024-03-24T11:20:00.555990+0000","last_active":"2024-03-24T11:20:17.724784+0000","last_peered":"2024-03-24T11:20:17.724784+0000","last_clean":"2024-03-24T11:20:17.724784+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:20:17.724784+0000","last_undegraded":"2024-03-24T11:20:17.724784+0000","last_fullsized":"2024-03-24T11:20:17.724784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.555913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.555913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:24.222390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.724932+0000","last_change":"2024-03-24T11:20:02.552338+0000","last_active":"2024-03-24T11:20:17.724932+0000","last_peered":"2024-03-24T11:20:17.724932+0000","last_clean":"2024-03-24T11:20:17.724932+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:20:17.724932+0000","last_undegraded":"2024-03-24T11:20:17.724932+0000","last_fullsized":"2024-03-24T11:20:17.724932+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:20:02.552224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.552224+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:21.289260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725112+0000","last_change":"2024-03-24T11:20:08.530821+0000","last_active":"2024-03-24T11:20:17.725112+0000","last_peered":"2024-03-24T11:20:17.725112+0000","last_clean":"2024-03-24T11:20:17.725112+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:20:17.725112+0000","last_undegraded":"2024-03-24T11:20:17.725112+0000","last_fullsized":"2024-03-24T11:20:17.725112+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:08.530740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:08.530740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:12.295185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726240+0000","last_change":"2024-03-24T11:20:17.382742+0000","last_active":"2024-03-24T11:20:17.726240+0000","last_peered":"2024-03-24T11:20:17.726240+0000","last_clean":"2024-03-24T11:20:17.726240+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:20:17.726240+0000","last_undegraded":"2024-03-24T11:20:17.726240+0000","last_fullsized":"2024-03-24T11:20:17.726240+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:17.382613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:17.382613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.038316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":308,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726982+0000","last_change":"2024-03-24T11:19:18.724688+0000","last_active":"2024-03-24T11:20:17.726982+0000","last_peered":"2024-03-24T11:20:17.726982+0000","last_clean":"2024-03-24T11:20:17.726982+0000","last_became_active":"2024-03-24T11:18:44.123994+0000","last_became_peered":"2024-03-24T11:18:44.123994+0000","last_unstale":"2024-03-24T11:20:17.726982+0000","last_undegraded":"2024-03-24T11:20:17.726982+0000","last_fullsized":"2024-03-24T11:20:17.726982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-24T11:19:18.724552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:19:18.724552+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:46.099214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727873+0000","last_change":"2024-03-24T11:20:05.392270+0000","last_active":"2024-03-24T11:20:17.727873+0000","last_peered":"2024-03-24T11:20:17.727873+0000","last_clean":"2024-03-24T11:20:17.727873+0000","last_became_active":"2024-03-24T11:18:53.678099+0000","last_became_peered":"2024-03-24T11:18:53.678099+0000","last_unstale":"2024-03-24T11:20:17.727873+0000","last_undegraded":"2024-03-24T11:20:17.727873+0000","last_fullsized":"2024-03-24T11:20:17.727873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.392195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.392195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:19.644580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727438+0000","last_change":"2024-03-24T11:18:56.141576+0000","last_active":"2024-03-24T11:20:17.727438+0000","last_peered":"2024-03-24T11:20:17.727438+0000","last_clean":"2024-03-24T11:20:17.727438+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:20:17.727438+0000","last_undegraded":"2024-03-24T11:20:17.727438+0000","last_fullsized":"2024-03-24T11:20:17.727438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.491233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725997+0000","last_change":"2024-03-24T11:20:10.834643+0000","last_active":"2024-03-24T11:20:17.725997+0000","last_peered":"2024-03-24T11:20:17.725997+0000","last_clean":"2024-03-24T11:20:17.725997+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:20:17.725997+0000","last_undegraded":"2024-03-24T11:20:17.725997+0000","last_fullsized":"2024-03-24T11:20:17.725997+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:10.834537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:10.834537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:23.214600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721802+0000","last_change":"2024-03-24T11:18:59.710726+0000","last_active":"2024-03-24T11:20:17.721802+0000","last_peered":"2024-03-24T11:20:17.721802+0000","last_clean":"2024-03-24T11:20:17.721802+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:20:17.721802+0000","last_undegraded":"2024-03-24T11:20:17.721802+0000","last_fullsized":"2024-03-24T11:20:17.721802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:19.585199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720709+0000","last_change":"2024-03-24T11:18:53.681515+0000","last_active":"2024-03-24T11:20:17.720709+0000","last_peered":"2024-03-24T11:20:17.720709+0000","last_clean":"2024-03-24T11:20:17.720709+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:20:17.720709+0000","last_undegraded":"2024-03-24T11:20:17.720709+0000","last_fullsized":"2024-03-24T11:20:17.720709+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":158,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:21.614060+0000","last_change":"2024-03-24T11:20:06.880339+0000","last_active":"2024-03-24T11:20:21.614060+0000","last_peered":"2024-03-24T11:20:21.614060+0000","last_clean":"2024-03-24T11:20:21.614060+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:20:21.614060+0000","last_undegraded":"2024-03-24T11:20:21.614060+0000","last_fullsized":"2024-03-24T11:20:21.614060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:20:06.880224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.880224+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:32.826706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721534+0000","last_change":"2024-03-24T11:18:55.681576+0000","last_active":"2024-03-24T11:20:17.721534+0000","last_peered":"2024-03-24T11:20:17.721534+0000","last_clean":"2024-03-24T11:20:17.721534+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:20:17.721534+0000","last_undegraded":"2024-03-24T11:20:17.721534+0000","last_fullsized":"2024-03-24T11:20:17.721534+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.955414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726622+0000","last_change":"2024-03-24T11:20:13.385604+0000","last_active":"2024-03-24T11:20:17.726622+0000","last_peered":"2024-03-24T11:20:17.726622+0000","last_clean":"2024-03-24T11:20:17.726622+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:20:17.726622+0000","last_undegraded":"2024-03-24T11:20:17.726622+0000","last_fullsized":"2024-03-24T11:20:17.726622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:13.385497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:13.385497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:18.985789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722118+0000","last_change":"2024-03-24T11:20:17.347069+0000","last_active":"2024-03-24T11:20:17.722118+0000","last_peered":"2024-03-24T11:20:17.722118+0000","last_clean":"2024-03-24T11:20:17.722118+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:20:17.722118+0000","last_undegraded":"2024-03-24T11:20:17.722118+0000","last_fullsized":"2024-03-24T11:20:17.722118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:17.346965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:17.346965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:44.870753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722018+0000","last_change":"2024-03-24T11:20:07.202667+0000","last_active":"2024-03-24T11:20:17.722018+0000","last_peered":"2024-03-24T11:20:17.722018+0000","last_clean":"2024-03-24T11:20:17.722018+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:20:17.722018+0000","last_undegraded":"2024-03-24T11:20:17.722018+0000","last_fullsized":"2024-03-24T11:20:17.722018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:07.202576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:07.202576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:10.625674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":87,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721344+0000","last_change":"2024-03-24T11:18:56.142025+0000","last_active":"2024-03-24T11:20:17.721344+0000","last_peered":"2024-03-24T11:20:17.721344+0000","last_clean":"2024-03-24T11:20:17.721344+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:20:17.721344+0000","last_undegraded":"2024-03-24T11:20:17.721344+0000","last_fullsized":"2024-03-24T11:20:17.721344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:23.374097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726350+0000","last_change":"2024-03-24T11:20:05.186178+0000","last_active":"2024-03-24T11:20:17.726350+0000","last_peered":"2024-03-24T11:20:17.726350+0000","last_clean":"2024-03-24T11:20:17.726350+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:20:17.726350+0000","last_undegraded":"2024-03-24T11:20:17.726350+0000","last_fullsized":"2024-03-24T11:20:17.726350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_clean_scrub_stamp":"2024-03-24T11:20:05.186078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:17.695962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720276+0000","last_change":"2024-03-24T11:18:59.714865+0000","last_active":"2024-03-24T11:20:17.720276+0000","last_peered":"2024-03-24T11:20:17.720276+0000","last_clean":"2024-03-24T11:20:17.720276+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:20:17.720276+0000","last_undegraded":"2024-03-24T11:20:17.720276+0000","last_fullsized":"2024-03-24T11:20:17.720276+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725326+0000","last_change":"2024-03-24T11:20:09.320186+0000","last_active":"2024-03-24T11:20:17.725326+0000","last_peered":"2024-03-24T11:20:17.725326+0000","last_clean":"2024-03-24T11:20:17.725326+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:20:17.725326+0000","last_undegraded":"2024-03-24T11:20:17.725326+0000","last_fullsized":"2024-03-24T11:20:17.725326+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:09.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:09.320059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:11.542154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.720896+0000","last_change":"2024-03-24T11:20:14.364904+0000","last_active":"2024-03-24T11:20:17.720896+0000","last_peered":"2024-03-24T11:20:17.720896+0000","last_clean":"2024-03-24T11:20:17.720896+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:20:17.720896+0000","last_undegraded":"2024-03-24T11:20:17.720896+0000","last_fullsized":"2024-03-24T11:20:17.720896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:20:14.364738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:14.364738+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:15.151256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721797+0000","last_change":"2024-03-24T11:20:01.039976+0000","last_active":"2024-03-24T11:20:17.721797+0000","last_peered":"2024-03-24T11:20:17.721797+0000","last_clean":"2024-03-24T11:20:17.721797+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:20:17.721797+0000","last_undegraded":"2024-03-24T11:20:17.721797+0000","last_fullsized":"2024-03-24T11:20:17.721797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:20:01.039839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:01.039839+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:01.168501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725621+0000","last_change":"2024-03-24T11:19:00.314552+0000","last_active":"2024-03-24T11:20:17.725621+0000","last_peered":"2024-03-24T11:20:17.725621+0000","last_clean":"2024-03-24T11:20:17.725621+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:20:17.725621+0000","last_undegraded":"2024-03-24T11:20:17.725621+0000","last_fullsized":"2024-03-24T11:20:17.725621+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.721666+0000","last_change":"2024-03-24T11:18:53.683466+0000","last_active":"2024-03-24T11:20:17.721666+0000","last_peered":"2024-03-24T11:20:17.721666+0000","last_clean":"2024-03-24T11:20:17.721666+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:20:17.721666+0000","last_undegraded":"2024-03-24T11:20:17.721666+0000","last_fullsized":"2024-03-24T11:20:17.721666+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717808+0000","last_change":"2024-03-24T11:20:14.019874+0000","last_active":"2024-03-24T11:20:17.717808+0000","last_peered":"2024-03-24T11:20:17.717808+0000","last_clean":"2024-03-24T11:20:17.717808+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:20:17.717808+0000","last_undegraded":"2024-03-24T11:20:17.717808+0000","last_fullsized":"2024-03-24T11:20:17.717808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-03-24T11:20:14.019717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:14.019717+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:42.126712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727799+0000","last_change":"2024-03-24T11:18:57.697565+0000","last_active":"2024-03-24T11:20:17.727799+0000","last_peered":"2024-03-24T11:20:17.727799+0000","last_clean":"2024-03-24T11:20:17.727799+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:20:17.727799+0000","last_undegraded":"2024-03-24T11:20:17.727799+0000","last_fullsized":"2024-03-24T11:20:17.727799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.519845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717952+0000","last_change":"2024-03-24T11:18:59.706057+0000","last_active":"2024-03-24T11:20:17.717952+0000","last_peered":"2024-03-24T11:20:17.717952+0000","last_clean":"2024-03-24T11:20:17.717952+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:20:17.717952+0000","last_undegraded":"2024-03-24T11:20:17.717952+0000","last_fullsized":"2024-03-24T11:20:17.717952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:28.016361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728503+0000","last_change":"2024-03-24T11:18:53.679203+0000","last_active":"2024-03-24T11:20:17.728503+0000","last_peered":"2024-03-24T11:20:17.728503+0000","last_clean":"2024-03-24T11:20:17.728503+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:20:17.728503+0000","last_undegraded":"2024-03-24T11:20:17.728503+0000","last_fullsized":"2024-03-24T11:20:17.728503+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":83,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727648+0000","last_change":"2024-03-24T11:18:56.141573+0000","last_active":"2024-03-24T11:20:17.727648+0000","last_peered":"2024-03-24T11:20:17.727648+0000","last_clean":"2024-03-24T11:20:17.727648+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:20:17.727648+0000","last_undegraded":"2024-03-24T11:20:17.727648+0000","last_fullsized":"2024-03-24T11:20:17.727648+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:21.850624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717620+0000","last_change":"2024-03-24T11:20:00.029193+0000","last_active":"2024-03-24T11:20:17.717620+0000","last_peered":"2024-03-24T11:20:17.717620+0000","last_clean":"2024-03-24T11:20:17.717620+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:20:17.717620+0000","last_undegraded":"2024-03-24T11:20:17.717620+0000","last_fullsized":"2024-03-24T11:20:17.717620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:00.029091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:00.029091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.480294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725049+0000","last_change":"2024-03-24T11:18:59.715347+0000","last_active":"2024-03-24T11:20:17.725049+0000","last_peered":"2024-03-24T11:20:17.725049+0000","last_clean":"2024-03-24T11:20:17.725049+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:20:17.725049+0000","last_undegraded":"2024-03-24T11:20:17.725049+0000","last_fullsized":"2024-03-24T11:20:17.725049+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.582227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718824+0000","last_change":"2024-03-24T11:18:57.699380+0000","last_active":"2024-03-24T11:20:17.718824+0000","last_peered":"2024-03-24T11:20:17.718824+0000","last_clean":"2024-03-24T11:20:17.718824+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:20:17.718824+0000","last_undegraded":"2024-03-24T11:20:17.718824+0000","last_fullsized":"2024-03-24T11:20:17.718824+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727641+0000","last_change":"2024-03-24T11:20:02.414997+0000","last_active":"2024-03-24T11:20:17.727641+0000","last_peered":"2024-03-24T11:20:17.727641+0000","last_clean":"2024-03-24T11:20:17.727641+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:20:17.727641+0000","last_undegraded":"2024-03-24T11:20:17.727641+0000","last_fullsized":"2024-03-24T11:20:17.727641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:02.414845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:02.414845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:06.613098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":76,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722633+0000","last_change":"2024-03-24T11:18:55.686410+0000","last_active":"2024-03-24T11:20:17.722633+0000","last_peered":"2024-03-24T11:20:17.722633+0000","last_clean":"2024-03-24T11:20:17.722633+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:20:17.722633+0000","last_undegraded":"2024-03-24T11:20:17.722633+0000","last_fullsized":"2024-03-24T11:20:17.722633+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:20.066223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.727990+0000","last_change":"2024-03-24T11:19:00.312496+0000","last_active":"2024-03-24T11:20:17.727990+0000","last_peered":"2024-03-24T11:20:17.727990+0000","last_clean":"2024-03-24T11:20:17.727990+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:20:17.727990+0000","last_undegraded":"2024-03-24T11:20:17.727990+0000","last_fullsized":"2024-03-24T11:20:17.727990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:18:58.683079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726542+0000","last_change":"2024-03-24T11:20:15.381720+0000","last_active":"2024-03-24T11:20:17.726542+0000","last_peered":"2024-03-24T11:20:17.726542+0000","last_clean":"2024-03-24T11:20:17.726542+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:20:17.726542+0000","last_undegraded":"2024-03-24T11:20:17.726542+0000","last_fullsized":"2024-03-24T11:20:17.726542+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:15.381636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:15.381636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:32.306821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.717707+0000","last_change":"2024-03-24T11:18:53.679465+0000","last_active":"2024-03-24T11:20:17.717707+0000","last_peered":"2024-03-24T11:20:17.717707+0000","last_clean":"2024-03-24T11:20:17.717707+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:20:17.717707+0000","last_undegraded":"2024-03-24T11:20:17.717707+0000","last_fullsized":"2024-03-24T11:20:17.717707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:22.456813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:19.106711+0000","last_change":"2024-03-24T11:20:19.106754+0000","last_active":"2024-03-24T11:20:19.106711+0000","last_peered":"2024-03-24T11:20:19.106711+0000","last_clean":"2024-03-24T11:20:19.106711+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:20:19.106711+0000","last_undegraded":"2024-03-24T11:20:19.106711+0000","last_fullsized":"2024-03-24T11:20:19.106711+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:20:19.106649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:19.106649+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:38.089530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725356+0000","last_change":"2024-03-24T11:20:06.548123+0000","last_active":"2024-03-24T11:20:17.725356+0000","last_peered":"2024-03-24T11:20:17.725356+0000","last_clean":"2024-03-24T11:20:17.725356+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:20:17.725356+0000","last_undegraded":"2024-03-24T11:20:17.725356+0000","last_fullsized":"2024-03-24T11:20:17.725356+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:06.548036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:06.548036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:08.537924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.722247+0000","last_change":"2024-03-24T11:20:04.159022+0000","last_active":"2024-03-24T11:20:17.722247+0000","last_peered":"2024-03-24T11:20:17.722247+0000","last_clean":"2024-03-24T11:20:17.722247+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:20:17.722247+0000","last_undegraded":"2024-03-24T11:20:17.722247+0000","last_fullsized":"2024-03-24T11:20:17.722247+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:04.158945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:20:04.158945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:13.212380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.726123+0000","last_change":"2024-03-24T11:20:03.233345+0000","last_active":"2024-03-24T11:20:17.726123+0000","last_peered":"2024-03-24T11:20:17.726123+0000","last_clean":"2024-03-24T11:20:17.726123+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:20:17.726123+0000","last_undegraded":"2024-03-24T11:20:17.726123+0000","last_fullsized":"2024-03-24T11:20:17.726123+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:20:03.233200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:20:03.233200+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:29.734595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":103,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718574+0000","last_change":"2024-03-24T11:19:18.829942+0000","last_active":"2024-03-24T11:20:17.718574+0000","last_peered":"2024-03-24T11:20:17.718574+0000","last_clean":"2024-03-24T11:20:17.718574+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:20:17.718574+0000","last_undegraded":"2024-03-24T11:20:17.718574+0000","last_fullsized":"2024-03-24T11:20:17.718574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718823+0000","last_change":"2024-03-24T11:20:16.026203+0000","last_active":"2024-03-24T11:20:17.718823+0000","last_peered":"2024-03-24T11:20:17.718823+0000","last_clean":"2024-03-24T11:20:17.718823+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:20:17.718823+0000","last_undegraded":"2024-03-24T11:20:17.718823+0000","last_fullsized":"2024-03-24T11:20:17.718823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:20:16.026113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:21:38.997969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.725137+0000","last_change":"2024-03-24T11:18:57.701581+0000","last_active":"2024-03-24T11:20:17.725137+0000","last_peered":"2024-03-24T11:20:17.725137+0000","last_clean":"2024-03-24T11:20:17.725137+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:20:17.725137+0000","last_undegraded":"2024-03-24T11:20:17.725137+0000","last_fullsized":"2024-03-24T11:20:17.725137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:18:56.669278+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.728023+0000","last_change":"2024-03-24T11:18:53.685545+0000","last_active":"2024-03-24T11:20:17.728023+0000","last_peered":"2024-03-24T11:20:17.728023+0000","last_clean":"2024-03-24T11:20:17.728023+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:20:17.728023+0000","last_undegraded":"2024-03-24T11:20:17.728023+0000","last_fullsized":"2024-03-24T11:20:17.728023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:18:52.658804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T11:20:17.718394+0000","last_change":"2024-03-24T11:18:55.680886+0000","last_active":"2024-03-24T11:20:17.718394+0000","last_peered":"2024-03-24T11:20:17.718394+0000","last_clean":"2024-03-24T11:20:17.718394+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:20:17.718394+0000","last_undegraded":"2024-03-24T11:20:17.718394+0000","last_fullsized":"2024-03-24T11:20:17.718394+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:18:54.665517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070486,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32516,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712636,"statfs":{"total":95995035648,"available":95961739264,"internally_reserved":0,"allocated":1372160,"data_stored":963547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364827,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1359872,"data_stored":963768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31964,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713188,"statfs":{"total":95995035648,"available":95962304512,"internally_reserved":0,"allocated":806912,"data_stored":374237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1380352,"data_stored":963302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":774144,"data_stored":372695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":373429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":790528,"data_stored":373607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31928,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713224,"statfs":{"total":95995035648,"available":95962341376,"internally_reserved":0,"allocated":770048,"data_stored":372961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:20:29.153 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:20:29.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058430077Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058619277Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058636277Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058642734Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.05864835Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058654252Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058659715Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058665031Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T11:20:29.310 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058670333Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058675433Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058680336Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058685198Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.0586902Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=settings t=2024-03-24T11:20:29.058695397Z level=info msg="App mode production" 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore t=2024-03-24T11:20:29.058768691Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore t=2024-03-24T11:20:29.058789736Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.059739753Z level=info msg="Starting DB migrations" 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.05992366Z level=info msg="Executing migration" id="create migration_log table" 2024-03-24T11:20:29.311 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.060439559Z level=info msg="Executing migration" id="create user table" 2024-03-24T11:20:29.312 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.060872683Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-24T11:20:29.312 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.061374367Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-24T11:20:29.312 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.061848723Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-24T11:20:29.312 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.062451461Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-24T11:20:29.312 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.062882446Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-24T11:20:29.312 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.063579027Z level=info msg="Executing migration" id="create user table v2" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.064002722Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.064414612Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.064901318Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.065325471Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.065682808Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.066165263Z level=info msg="Executing migration" id="Update user table charset" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.066460219Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.066907959Z level=info msg="Executing migration" id="Add missing user data" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.06728686Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.067828572Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.068378071Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.068933542Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.070408474Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-24T11:20:29.313 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.070855113Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.07132491Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.07179311Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.072200265Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.072590121Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.072895628Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.073331311Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.073702723Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.074108071Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.074462801Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.07509933Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.075493288Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.075902574Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.076332697Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-24T11:20:29.314 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.07669981Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.077129103Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.077436203Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.077764857Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.078101345Z level=info msg="Executing migration" id="create star table" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.078445649Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.078860703Z level=info msg="Executing migration" id="create org table v1" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.079251256Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.079655546Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.080043125Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.080443673Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.080844033Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.081254409Z level=info msg="Executing migration" id="Update org table charset" 2024-03-24T11:20:29.315 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.081517164Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.081786175Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.082101253Z level=info msg="Executing migration" id="create dashboard table" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.082481852Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.082879789Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.083285008Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.083705484Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.084501419Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.085463349Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.086362394Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.086752604Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.087623436Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.088453527Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.088803679Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-24T11:20:29.316 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.089257687Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.089518056Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.092216422Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.094868939Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.097539314Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.098048069Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.100640887Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.103405423Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.105966464Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.106284199Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.106565495Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.107092718Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.109927653Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.112684448Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-24T11:20:29.317 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.113256992Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.114059856Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.114524733Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.114943845Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.115921333Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.11645926Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.117315774Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.118317766Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.119103329Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.119533125Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.120381394Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.121327774Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.121979307Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-24T11:20:29.318 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.122490848Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.123354146Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.124221375Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.125133444Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.125712866Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.126565682Z level=info msg="Executing migration" id="create data_source table" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.127496494Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.128357019Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-24T11:20:29.319 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.129140576Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-24T11:20:29.320 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.129990608Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T11:20:29.320 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.130941134Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-24T11:20:29.320 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.132159282Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-24T11:20:29.320 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.132586717Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-24T11:20:29.320 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.133439217Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-24T11:20:29.320 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.13440402Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-24T11:20:29.320 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.135319674Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.135942495Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.136825462Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.137702558Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.138654193Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.13935987Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.142156029Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.142500233Z level=info msg="Executing migration" id="Add uid column" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.145005256Z level=info msg="Executing migration" id="Update uid value" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.147944715Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.148495449Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.151372789Z level=info msg="Executing migration" id="create api_key table" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.154164009Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.156885277Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-24T11:20:29.321 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.157369277Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.16039649Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.163105624Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.165986887Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.168787866Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.171314679Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.174201403Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.176841489Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.179567479Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.182173413Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.184866779Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.187564973Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.190275508Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.192944362Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-24T11:20:29.322 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.195717212Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.198261831Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.200968515Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.203768268Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.206334726Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.208851176Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.211517995Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.214004928Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.216703915Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.219387984Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.222192879Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.224656763Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.225542685Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.226393091Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T11:20:29.323 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.227239283Z level=info msg="Executing migration" id="create quota table v1" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.228114261Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.228884697Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.229854825Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.230557417Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.231468021Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.232386819Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.233128854Z level=info msg="Executing migration" id="create session table" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.23397061Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.234808885Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.23577339Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.23664043Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-24T11:20:29.324 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.237386299Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.238297396Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.238759864Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.239099941Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.239921481Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.240356468Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.240846385Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.241458837Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.241760939Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.242352888Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.242956877Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.243842198Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.24483754Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.245593557Z level=info msg="Executing migration" id="create alert table v1" 2024-03-24T11:20:29.325 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.246577708Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.247308865Z level=info msg="Executing migration" id="add index alert state" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.248315369Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.249014445Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.24986578Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.25084777Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.251567956Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.254510339Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.257440732Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.260129359Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.262944034Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.265836611Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.268746924Z level=info msg="Executing migration" id="Add column is_default" 2024-03-24T11:20:29.326 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.271470561Z level=info msg="Executing migration" id="Add column frequency" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.274187153Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.276947893Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.279613807Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.282647242Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.285551396Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.288421503Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.291131125Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.293961849Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.297023332Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.299798021Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.302292216Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.304952903Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.307634384Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-24T11:20:29.327 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.310050666Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-24T11:20:29.328 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.312707289Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-24T11:20:29.328 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.31542783Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-24T11:20:29.406 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:29.560 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.318039369Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.320618094Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.323410031Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.326125525Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.32889884Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.331567087Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.334202529Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.336837336Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.337260016Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.337692148Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.340535843Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-24T11:20:29.561 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.343234991Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.343941803Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.346649226Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.349357554Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.351920408Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.354703186Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.357184178Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.360017422Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.360819014Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.361510429Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.362298255Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.362675692Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.363142815Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.363905267Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-24T11:20:29.562 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.364739884Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.36551264Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.366485963Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.367170556Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.367976716Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.368936567Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.369717584Z level=info msg="Executing migration" id="Move region to single row" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.37039975Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.371166682Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.371602743Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.37208265Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T11:20:29.563 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.372874421Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.373742844Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.374531512Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.375349265Z level=info msg="Executing migration" id="create test_data table" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.37616257Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.376584321Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.377401268Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.378216522Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.379065095Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.379915865Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.380838634Z level=info msg="Executing migration" id="create team table" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.381786766Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-24T11:20:29.565 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.382542606Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.38337177Z level=info msg="Executing migration" id="create team member table" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.384250341Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.385056501Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.3858939Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.386679673Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.387570913Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.390496049Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.393085678Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.395930673Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.398634821Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.401148741Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.401601318Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.404372058Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-24T11:20:29.566 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.406956954Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.409716756Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.412577681Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.41313574Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.41593164Z level=info msg="Executing migration" id="create tag table" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.418679772Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.421572793Z level=info msg="Executing migration" id="create login attempt table" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.421991882Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.424908185Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.427451389Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.43016783Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.430624749Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.433342446Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-24T11:20:29.567 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.436056923Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.438648534Z level=info msg="Executing migration" id="create user auth table" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.439067226Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.441751269Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.44456729Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.447249437Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.448139953Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.45091577Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.453570312Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.456127607Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.458778106Z level=info msg="Executing migration" id="create server_lock table" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.461327602Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.463963967Z level=info msg="Executing migration" id="create user auth token table" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.466584629Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-24T11:20:29.568 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.467034842Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.46985383Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.470632815Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.471458802Z level=info msg="Executing migration" id="create cache_data table" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.471914894Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.472853314Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.473505213Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.47433796Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.474653947Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.474976981Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.475442573Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.47591594Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.476353743Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.476628874Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-24T11:20:29.569 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.477055717Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.477483015Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.477927337Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.478366658Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.479206007Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.479635534Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.47995182Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.480354968Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.48077474Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.481206304Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.481478669Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.481912073Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-24T11:20:29.570 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.482322674Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.482743006Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.483184976Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.483998701Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.484417373Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.484870977Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.488251325Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.491610584Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.492108861Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.492534092Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.493383718Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.493827993Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.494255617Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-24T11:20:29.571 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.494665609Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.495089859Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.495374625Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.49621733Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.497042526Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.497830259Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.498243359Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.498632442Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.499449731Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.500275002Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.500718334Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.501539344Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.502369519Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-24T11:20:29.572 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.502673247Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.503116405Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.50582877Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.508355037Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.511055195Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.511967788Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.514772887Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.517371876Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.519920047Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-24T11:20:29.573 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.520762211Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.523596993Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.524282902Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.525128865Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.525419226Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.526336942Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.52715078Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.528025526Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.528467959Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.529304257Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.530221351Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.531037656Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-24T11:20:29.574 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.53147772Z level=info msg="Executing migration" id="create alert_image table" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.532321387Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.533156812Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.534103613Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.53474195Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.535591978Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.535664076Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.536450351Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.537389626Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.537884083Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.537997734Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.538103076Z level=info msg="alerts found to migrate" alerts=0 2024-03-24T11:20:29.575 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.538923171Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.539607144Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.540603787Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.541148626Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.542037119Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.542860393Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.543695632Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.544659949Z level=info msg="Executing migration" id="create data_keys table" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.545384904Z level=info msg="Executing migration" id="create secrets table" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.546233867Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.550541678Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.551456357Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-24T11:20:29.576 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.552323779Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-24T11:20:29.577 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.556522333Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-24T11:20:29.577 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.560583549Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-24T11:20:29.577 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.561138769Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-24T11:20:29.577 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.564081971Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T11:20:29.805 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: 5.17 scrub starts 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: 5.17 scrub ok 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[29608]: Deploying daemon node-exporter.a on smithi179 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: 5.17 scrub starts 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: 5.17 scrub ok 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:29 smithi179 ceph-mon[37907]: Deploying daemon node-exporter.a on smithi179 2024-03-24T11:20:29.810 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: 5.17 scrub starts 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: 5.17 scrub ok 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:29.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-mon[33998]: Deploying daemon node-exporter.a on smithi179 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.56701118Z level=info msg="Executing migration" id="create permission table" 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.56976757Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.570354001Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.573248392Z level=info msg="Executing migration" id="create role table" 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.576001495Z level=info msg="Executing migration" id="add column display_name" 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.578830225Z level=info msg="Executing migration" id="add column group_name" 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.579771878Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-24T11:20:29.812 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.582659205Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.58540348Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.58846903Z level=info msg="Executing migration" id="create team role table" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.588897673Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.591833596Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.594668314Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.59730156Z level=info msg="Executing migration" id="create user role table" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.597766046Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.600559228Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.60334617Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.605954403Z level=info msg="Executing migration" id="create builtin role table" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.606372005Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.609164368Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.611929774Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-24T11:20:29.813 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.614678711Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.615279359Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.618076201Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.620803215Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.623396757Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.623862508Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.626596085Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.629262379Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.631848439Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.632279912Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.635019011Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.6356316Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.637750633Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-24T11:20:29.814 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.638763503Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-24T11:20:29.815 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.639205692Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-24T11:20:29.815 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.64001312Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-24T11:20:29.815 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.640446533Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-24T11:20:29.815 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.640904002Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-24T11:20:29.815 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.64190269Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.642217327Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.643120504Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.644033608Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.64485467Z level=info msg="Executing migration" id="add correlation config column" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.645876713Z level=info msg="Executing migration" id="create entity_events table" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.646887686Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.647574049Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.647664986Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.648589201Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.648762536Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.649044284Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.649960139Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-24T11:20:29.816 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.650914056Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.651803322Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.652431667Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.65341748Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.654179767Z level=info msg="Executing migration" id="Drop public config table" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.655062782Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.655506163Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.656378695Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.657246532Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.658221868Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.66133203Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.662365299Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.663393776Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.664259505Z level=info msg="Executing migration" id="add share column" 2024-03-24T11:20:29.817 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.66532508Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.666167047Z level=info msg="Executing migration" id="create file table" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.667036301Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.668033663Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.668524969Z level=info msg="Executing migration" id="create file_meta table" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.671362106Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.674265567Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.67696334Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.677350411Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.680242668Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.680708712Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.683087427Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.685859845Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-24T11:20:29.818 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.6864023Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.689359503Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.692294216Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.694949694Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.696019431Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.698850715Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.701647858Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.711264Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.711747402Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.712814346Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.716505577Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-24T11:20:29.819 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.717252047Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.717617814Z level=info msg="Executing migration" id="create folder table" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.718562189Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.719341767Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.720172374Z level=info msg="Executing migration" id="Update folder title length" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.720494179Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=migrator t=2024-03-24T11:20:29.721267326Z level=info msg="migrations completed" performed=485 skipped=0 duration=661.374364ms 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore t=2024-03-24T11:20:29.721689042Z level=info msg="Created default organization" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=plugin.loader t=2024-03-24T11:20:29.777685509Z level=info msg="Plugin registered" pluginID=input 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=plugin.loader t=2024-03-24T11:20:29.804707023Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=plugin.loader t=2024-03-24T11:20:29.804737703Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=secrets t=2024-03-24T11:20:29.804806415Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=query_data t=2024-03-24T11:20:29.808660196Z level=info msg="Query Service initialization" 2024-03-24T11:20:29.820 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=live.push_http t=2024-03-24T11:20:29.811395134Z level=info msg="Live Push Gateway initialization" 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=infra.usagestats.collector t=2024-03-24T11:20:30.455414095Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=provisioning.datasources t=2024-03-24T11:20:30.456733054Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=provisioning.datasources t=2024-03-24T11:20:30.468043701Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=provisioning.alerting t=2024-03-24T11:20:30.479851022Z level=info msg="starting to provision alerting" 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=provisioning.alerting t=2024-03-24T11:20:30.479877193Z level=info msg="finished to provision alerting" 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=grafanaStorageLogger t=2024-03-24T11:20:30.480719468Z level=info msg="storage starting" 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=ngalert.state.manager t=2024-03-24T11:20:30.480878496Z level=info msg="Warming state cache for startup" 2024-03-24T11:20:30.652 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=http.server t=2024-03-24T11:20:30.482585628Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T11:20:30.653 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=ngalert.state.manager t=2024-03-24T11:20:30.483012332Z level=info msg="State cache has been initialized" states=0 duration=2.132782ms 2024-03-24T11:20:30.653 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=ticker t=2024-03-24T11:20:30.483068159Z level=info msg=starting first_tick=2024-03-24T11:20:40Z 2024-03-24T11:20:30.653 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=ngalert.multiorg.alertmanager t=2024-03-24T11:20:30.483079275Z level=info msg="starting MultiOrg Alertmanager" 2024-03-24T11:20:30.653 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore.transactions t=2024-03-24T11:20:30.4944904Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:20:30.653 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore.transactions t=2024-03-24T11:20:30.494632893Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:20:30.653 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore.transactions t=2024-03-24T11:20:30.505646639Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-24T11:20:31.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:20:31.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:20:31.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[37907]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:31.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:20:31.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:20:31.130 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:20:31.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:20:31.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[29608]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:31.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:20:31.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:30 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:20:31.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:20:31.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:20:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-mon[33998]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:20:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:30 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:20:31.317 INFO:teuthology.orchestra.run.smithi179.stdout:epoch 89 2024-03-24T11:20:31.317 INFO:teuthology.orchestra.run.smithi179.stdout:fsid 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:20:31.317 INFO:teuthology.orchestra.run.smithi179.stdout:created 2024-03-24T11:13:18.639983+0000 2024-03-24T11:20:31.317 INFO:teuthology.orchestra.run.smithi179.stdout:modified 2024-03-24T11:20:28.761784+0000 2024-03-24T11:20:31.317 INFO:teuthology.orchestra.run.smithi179.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:crush_version 18 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:full_ratio 0.95 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:backfillfull_ratio 0.9 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:nearfull_ratio 0.85 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:require_min_compat_client luminous 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:min_compat_client luminous 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:require_osd_release squid 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:stretch_mode_enabled false 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 69 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-24T11:20:31.318 INFO:teuthology.orchestra.run.smithi179.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:max_osd 8 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377] [v2:172.21.15.179:6804/1725503377,v1:172.21.15.179:6805/1725503377] exists,up 14f300f0-7498-4ead-a87e-d9f38bdff21b 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275] [v2:172.21.15.179:6812/2564847275,v1:172.21.15.179:6813/2564847275] exists,up 3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842] [v2:172.21.15.179:6820/3297713842,v1:172.21.15.179:6821/3297713842] exists,up 863b92b8-c68c-41a3-aed4-fd79e96d34dc 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3 up in weight 1 up_from 26 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471] [v2:172.21.15.179:6828/944142471,v1:172.21.15.179:6829/944142471] exists,up c3d6b564-195d-4771-9aae-93a9e2eaba52 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4 up in weight 1 up_from 31 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415] [v2:172.21.15.181:6802/3231660415,v1:172.21.15.181:6803/3231660415] exists,up 91bf5303-3a94-407d-aef1-e43bf0e16176 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5 up in weight 1 up_from 41 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782] [v2:172.21.15.181:6810/775575782,v1:172.21.15.181:6811/775575782] exists,up cba3f489-e8be-46f7-a281-e69a0ad08646 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6 up in weight 1 up_from 50 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374] [v2:172.21.15.181:6818/2357509374,v1:172.21.15.181:6819/2357509374] exists,up 44a2c3a0-b704-4c10-a257-c54de47ee793 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7 up in weight 1 up_from 59 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117] [v2:172.21.15.181:6826/732039117,v1:172.21.15.181:6827/732039117] exists,up df60c4b4-d2f2-4e1f-8e1f-3169c403de00 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:pg_upmap_items 4.7 [1,2] 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:pg_upmap_items 4.d [1,5] 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:pg_upmap_items 4.18 [1,2] 2024-03-24T11:20:31.319 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/1519226585 expires 2024-03-25T11:19:31.961858+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:6800/2865046672 expires 2024-03-25T11:19:31.961858+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/2472348284 expires 2024-03-25T11:19:31.961858+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/2047439415 expires 2024-03-25T11:13:51.373673+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:6801/3169437 expires 2024-03-25T11:13:51.373673+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:6800/3169437 expires 2024-03-25T11:13:51.373673+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:6801/243537818 expires 2024-03-25T11:14:17.688945+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/3185072343 expires 2024-03-25T11:19:31.961858+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/3344305045 expires 2024-03-25T11:13:51.373673+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/2748363950 expires 2024-03-25T11:14:17.688945+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:6800/243537818 expires 2024-03-25T11:14:17.688945+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/2066930919 expires 2024-03-25T11:14:17.688945+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/2108875546 expires 2024-03-25T11:14:17.688945+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:6801/2865046672 expires 2024-03-25T11:19:31.961858+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/4087802319 expires 2024-03-25T11:19:31.961858+0000 2024-03-24T11:20:31.320 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/883221702 expires 2024-03-25T11:13:51.373673+0000 2024-03-24T11:20:31.321 INFO:teuthology.orchestra.run.smithi179.stdout:blocklist 172.21.15.179:0/4255117010 expires 2024-03-25T11:19:31.961858+0000 2024-03-24T11:20:31.709 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:31 smithi179 systemd[1]: Starting Ceph node-exporter.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:20:32.046 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:20:32.046 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 5.1a deep-scrub starts 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 5.1a deep-scrub ok 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2240065168' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:20:32.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 5.1a deep-scrub starts 2024-03-24T11:20:32.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 5.1a deep-scrub ok 2024-03-24T11:20:32.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2240065168' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:20:32.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:20:32.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:31 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:20:32.077 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6802/1725503377,v1:172.21.15.179:6803/1725503377] [v2:172.21.15.179:6804/1725503377,v1:172.21.15.179:6805/1725503377] exists,up 14f300f0-7498-4ead-a87e-d9f38bdff21b', 'osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6810/2564847275,v1:172.21.15.179:6811/2564847275] [v2:172.21.15.179:6812/2564847275,v1:172.21.15.179:6813/2564847275] exists,up 3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667', 'osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6818/3297713842,v1:172.21.15.179:6819/3297713842] [v2:172.21.15.179:6820/3297713842,v1:172.21.15.179:6821/3297713842] exists,up 863b92b8-c68c-41a3-aed4-fd79e96d34dc', 'osd.3 up in weight 1 up_from 26 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6826/944142471,v1:172.21.15.179:6827/944142471] [v2:172.21.15.179:6828/944142471,v1:172.21.15.179:6829/944142471] exists,up c3d6b564-195d-4771-9aae-93a9e2eaba52', 'osd.4 up in weight 1 up_from 31 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/3231660415,v1:172.21.15.181:6801/3231660415] [v2:172.21.15.181:6802/3231660415,v1:172.21.15.181:6803/3231660415] exists,up 91bf5303-3a94-407d-aef1-e43bf0e16176', 'osd.5 up in weight 1 up_from 41 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/775575782,v1:172.21.15.181:6809/775575782] [v2:172.21.15.181:6810/775575782,v1:172.21.15.181:6811/775575782] exists,up cba3f489-e8be-46f7-a281-e69a0ad08646', 'osd.6 up in weight 1 up_from 50 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/2357509374,v1:172.21.15.181:6817/2357509374] [v2:172.21.15.181:6818/2357509374,v1:172.21.15.181:6819/2357509374] exists,up 44a2c3a0-b704-4c10-a257-c54de47ee793', 'osd.7 up in weight 1 up_from 59 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/732039117,v1:172.21.15.181:6825/732039117] [v2:172.21.15.181:6826/732039117,v1:172.21.15.181:6827/732039117] exists,up df60c4b4-d2f2-4e1f-8e1f-3169c403de00'] 2024-03-24T11:20:32.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon mon.a config show 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 5.1a deep-scrub starts 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 5.1a deep-scrub ok 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2240065168' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:20:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:31 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:20:32.348 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:32.588 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:32 smithi179 bash[73895]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T11:20:32.959 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:32 smithi179 bash[73895]: Getting image source signatures 2024-03-24T11:20:32.959 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:32 smithi179 bash[73895]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T11:20:32.959 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:32 smithi179 bash[73895]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T11:20:32.959 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:32 smithi179 bash[73895]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 4.f deep-scrub starts 2024-03-24T11:20:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[29608]: 4.f deep-scrub ok 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 4.f deep-scrub starts 2024-03-24T11:20:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:32 smithi179 ceph-mon[37907]: 4.f deep-scrub ok 2024-03-24T11:20:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:20:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:20:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:20:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:20:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:33.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:33.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:33.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:20:33.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:20:33.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 4.f deep-scrub starts 2024-03-24T11:20:33.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:32 smithi181 ceph-mon[33998]: 4.f deep-scrub ok 2024-03-24T11:20:33.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:20:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:20:34.037 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[29608]: 3.12 scrub starts 2024-03-24T11:20:34.037 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[29608]: 3.12 scrub ok 2024-03-24T11:20:34.037 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:34.037 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:20:34.037 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:20:34.037 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[37907]: 3.12 scrub starts 2024-03-24T11:20:34.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[37907]: 3.12 scrub ok 2024-03-24T11:20:34.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:34.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:20:34.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:33 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:20:34.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:33 smithi181 ceph-mon[33998]: 3.12 scrub starts 2024-03-24T11:20:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:33 smithi181 ceph-mon[33998]: 3.12 scrub ok 2024-03-24T11:20:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:33 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:33 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:20:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:33 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:20:34.709 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:34 smithi179 bash[73895]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:20:35.004 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:34 smithi179 bash[73895]: Writing manifest to image destination 2024-03-24T11:20:35.127 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:20:35.127 INFO:teuthology.orchestra.run.smithi179.stdout: "name": "mon.a", 2024-03-24T11:20:35.127 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster": "ceph", 2024-03-24T11:20:35.127 INFO:teuthology.orchestra.run.smithi179.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-24T11:20:35.127 INFO:teuthology.orchestra.run.smithi179.stdout: "admin_socket_mode": "", 2024-03-24T11:20:35.127 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_client_required": "cephx, none", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_cluster_required": "cephx", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_debug": "true", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_service_required": "cephx", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "auth_supported": "", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_aio": "true", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_aio_poll_ms": "250", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_aio_reap_max": "16", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_async_discard": "false", 2024-03-24T11:20:35.128 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_block_size": "4096", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_debug_aio": "false", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_debug_inflight_ios": "false", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_enable_discard": "false", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_flock_retry": "3", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_inject_crash": "0", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_ioring": "false", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_ioring_hipri": "false", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-24T11:20:35.129 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bdev_type": "", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_alloc_size": "1048576", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_allocator": "hybrid", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_buffered_io": "true", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_check_for_zeros": "false", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_compact_log_sync": "false", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-24T11:20:35.130 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_max_log_runway": "4194304", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_max_prefetch": "1048576", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_min_flush_size": "524288", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_min_log_runway": "1048576", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_replay_recovery": "false", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_sync_write": "false", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-24T11:20:35.131 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_allocation_from_file": "true", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_allocator": "hybrid", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_create": "true", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_db_create": "false", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_db_path": "", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_db_size": "0", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_path": "", 2024-03-24T11:20:35.132 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_preallocate_file": "false", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_size": "107374182400", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_wal_create": "false", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_wal_path": "", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_block_wal_size": "100663296", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_bluefs": "true", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T11:20:35.133 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_autotune": "true", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_size": "0", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_type": "2q", 2024-03-24T11:20:35.134 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_clone_cow": "true", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_mode": "none", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_csum_type": "crc32c", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-24T11:20:35.135 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_freelist": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-24T11:20:35.136 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_small_allocations": "0", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_default_buffered_read": "true", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_default_buffered_write": "false", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-24T11:20:35.137 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fail_eio": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_on_mount": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_on_umount": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-24T11:20:35.138 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_ignore_data_csum": "false", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_log_op_age": "5.000000", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_max_alloc_size": "0", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_max_blob_size": "0", 2024-03-24T11:20:35.139 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_max_deferred_txc": "32", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_min_alloc_size": "0", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_nid_prealloc": "1024", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_retry_disk_reads": "3", 2024-03-24T11:20:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_rocksdb_cf": "true", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_spdk_mem": "512", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-24T11:20:35.141 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_tracing": "false", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_zero_block_detection": "false", 2024-03-24T11:20:35.142 INFO:teuthology.orchestra.run.smithi179.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephx_cluster_require_signatures": "false", 2024-03-24T11:20:35.143 INFO:teuthology.orchestra.run.smithi179.stdout: "cephx_cluster_require_version": "2", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "cephx_require_signatures": "false", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "cephx_require_version": "2", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "cephx_service_require_signatures": "false", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "cephx_service_require_version": "2", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "cephx_sign_messages": "true", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "chdir": "", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_acl_type": "", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_asio_thread_count": "2", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_cache_mid": "0.750000", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_cache_size": "16384", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_caps_release_delay": "5", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_check_pool_perm": "true", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-24T11:20:35.144 INFO:teuthology.orchestra.run.smithi179.stdout: "client_debug_force_sync_read": "false", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_debug_getattr_caps": "false", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_debug_inject_tick_delay": "0", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_die_on_failed_remount": "false", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_dirsize_rbytes": "true", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_force_lazyio": "false", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_fs": "", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_inject_release_failure": "false", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_max_inline_size": "4096", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_mds_namespace": "", 2024-03-24T11:20:35.145 INFO:teuthology.orchestra.run.smithi179.stdout: "client_metadata": "", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_mount_gid": "-1", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_mount_timeout": "300", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_mount_uid": "-1", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_mountpoint": "/", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_notify_timeout": "10", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_oc": "true", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_oc_max_dirty": "104857600", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_oc_max_objects": "1000", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_oc_size": "209715200", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_oc_target_dirty": "8388608", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_permissions": "true", 2024-03-24T11:20:35.146 INFO:teuthology.orchestra.run.smithi179.stdout: "client_quota": "true", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_quota_df": "true", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_readahead_max_bytes": "0", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_readahead_max_periods": "4", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_readahead_min": "131072", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_reconnect_stale": "false", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_shutdown_timeout": "30", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_snapdir": ".snap", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_tick_interval": "1", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_trace": "", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_try_dentry_invalidate": "false", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_use_faked_inos": "false", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "client_use_random_mds": "false", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "clog_to_graylog": "false", 2024-03-24T11:20:35.147 INFO:teuthology.orchestra.run.smithi179.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "clog_to_graylog_port": "12201", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "clog_to_monitors": "default=true", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "clog_to_syslog": "false", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "clog_to_syslog_level": "info", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "-", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_network": "", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_network_interface": "", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "compressor_zlib_isal": "false", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "compressor_zlib_level": "5", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "compressor_zlib_winsize": "-15", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "compressor_zstd_level": "1", 2024-03-24T11:20:35.148 INFO:teuthology.orchestra.run.smithi179.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crimson_alien_op_num_threads": "6", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crimson_seastar_cpu_cores": "", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_location": "", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_location_hook": "", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_location_hook_timeout": "10", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "daemonize": "false", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "daos_pool": "tank", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-24T11:20:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_alienstore": "0/5", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_allow_any_pool_priority": "false", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_asok": "1/5", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_asok_assert_abort": "false", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_asserts_on_shutdown": "true", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_auth": "1/5", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_bdev": "1/3", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_bluefs": "1/5", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_bluestore": "1/5", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_buffer": "0/1", 2024-03-24T11:20:35.150 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_ceph_exporter": "1/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_cephfs_mirror": "0/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_cephsqlite": "0/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_client": "0/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_compressor": "1/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_context": "0/1", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_crimson_interrupt": "0/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_crush": "1/1", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_crypto": "1/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_cyanstore": "0/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_deliberately_leak_memory": "false", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_disable_randomized_ping": "false", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_dpdk": "1/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_eventtrace": "1/5", 2024-03-24T11:20:35.151 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_filer": "0/1", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_filestore": "1/3", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_finisher": "1/1", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_fuse": "1/5", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_heartbeat_testing_span": "0", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_heartbeatmap": "1/5", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_javaclient": "1/5", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_journal": "1/3", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_journaler": "0/5", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_kstore": "1/5", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_lockdep": "0/1", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mclock": "1/5", 2024-03-24T11:20:35.152 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mds": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mds_balancer": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mds_locker": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mds_log": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mds_log_expire": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mds_migrator": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_memstore": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mgr": "2/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mgrc": "1/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_mon": "20/20", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_monc": "0/10", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_ms": "1/1", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_none": "0/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_objclass": "0/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_objectcacher": "0/5", 2024-03-24T11:20:35.153 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_objecter": "0/1", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_optracker": "0/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_osd": "1/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_paxos": "20/20", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_perfcounter": "1/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_prioritycache": "1/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rados": "0/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rbd": "0/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rbd_mirror": "0/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rbd_pwl": "0/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rbd_replay": "0/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_refs": "0/0", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_reserver": "1/1", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rgw": "1/5", 2024-03-24T11:20:35.154 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rgw_access": "1/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rgw_datacache": "1/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rgw_dbstore": "1/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rgw_flight": "1/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rgw_sync": "1/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_rocksdb": "4/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore": "0/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_backref": "0/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_cache": "0/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_cleaner": "0/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_device": "0/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_epm": "0/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-24T11:20:35.155 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_journal": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_lba": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_odata": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_omap": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_onode": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_t": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_seastore_tm": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_striper": "0/1", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_test": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_throttle": "1/1", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_timer": "0/1", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_tp": "0/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "debug_trace": "1/5", 2024-03-24T11:20:35.156 INFO:teuthology.orchestra.run.smithi179.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "device_failure_prediction_mode": "none", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "err_to_graylog": "false", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "err_to_journald": "false", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "err_to_stderr": "true", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "err_to_syslog": "false", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "event_tracing": "false", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "exporter_addr": "0.0.0.0", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "exporter_http_port": "9926", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "exporter_prio_limit": "5", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-24T11:20:35.157 INFO:teuthology.orchestra.run.smithi179.stdout: "exporter_sort_metrics": "true", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "exporter_stats_period": "5", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "fake_statfs_for_testing": "0", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "fatal_signal_handlers": "true", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filer_max_purge_ops": "10", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filer_max_truncate_ops": "128", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_apply_finisher_threads": "1", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_blackhole": "false", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_btrfs_clone_range": "true", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_btrfs_snap": "true", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_caller_concurrency": "10", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_collect_device_partition_information": "true", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_commit_timeout": "600.000000", 2024-03-24T11:20:35.158 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_debug_inject_read_err": "false", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_debug_omap_check": "false", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_debug_verify_split": "false", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_dump_file": "", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_fadvise": "true", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_fail_eio": "true", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_fd_cache_shards": "16", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_fd_cache_size": "128", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_fiemap": "false", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_fiemap_threshold": "4096", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-24T11:20:35.159 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_inject_stall": "0", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_journal_parallel": "false", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_journal_trailing": "false", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_journal_writeahead": "false", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_kill_at": "0", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattrs": "0", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-24T11:20:35.160 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_xattr_value_size": "0", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_merge_threshold": "-10", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_odsync_write": "false", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_omap_backend": "rocksdb", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_omap_backend_path": "", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-24T11:20:35.161 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_op_thread_timeout": "60", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_op_threads": "2", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_punch_hole": "false", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-24T11:20:35.162 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_queue_max_ops": "50", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_seek_data_hole": "false", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_sloppy_crc": "false", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_splice": "false", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_split_multiple": "2", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_split_rand_factor": "20", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_update_to": "1000", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-24T11:20:35.163 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_enable": "true", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_xfs_extsize": "false", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "filestore_zfs_snap": "false", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "fio_dir": "/tmp/fio", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "fsid": "6430538c-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:20:35.164 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_allow_other": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_atomic_o_trunc": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_big_writes": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_debug": "false", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_default_permissions": "false", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_disable_pagecache": "false", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_max_write": "0", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_multithreaded": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_require_active_mds": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_set_user_groups": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_splice_move": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_splice_read": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_splice_write": "true", 2024-03-24T11:20:35.165 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "fuse_use_invalidate_cb": "true", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "gss_target_name": "ceph", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_file": "", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_inject_failure": "0", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_interval": "5", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "host": "smithi179", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-24T11:20:35.166 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "inject_early_sigterm": "false", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "jaeger_agent_port": "6799", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "jaeger_tracing_enable": "false", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_aio": "true", 2024-03-24T11:20:35.167 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_align_min_size": "65536", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_block_align": "true", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_block_size": "4096", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_dio": "true", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_discard": "false", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_force_aio": "false", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_ignore_corruption": "false", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_max_write_bytes": "10485760", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_max_write_entries": "100", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_replay_from": "0", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-24T11:20:35.168 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_write_header_frequency": "0", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "journal_zero_on_create": "false", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "journaler_prefetch_periods": "10", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "journaler_prezero_periods": "5", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "journaler_write_head_interval": "15", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "key": "", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "keyfile": "", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_backend": "rocksdb", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_default_stripe_size": "65536", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_fsck_on_mount": "false", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_max_bytes": "67108864", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_max_ops": "512", 2024-03-24T11:20:35.169 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_nid_prealloc": "1024", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_onode_map_size": "1024", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_sync_submit_transaction": "false", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "kstore_sync_transaction": "false", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "librados_thread_count": "2", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "lockdep": "false", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "lockdep_force_backtrace": "false", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "log_coarse_timestamps": "true", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "log_flush_on_exit": "false", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "log_graylog_host": "127.0.0.1", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "log_graylog_port": "12201", 2024-03-24T11:20:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: "log_max_new": "1000", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_max_recent": "10000", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_stderr_prefix": "", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_stop_at_utilization": "0.970000", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_to_file": "true", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_to_graylog": "false", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_to_journald": "false", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_to_stderr": "false", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "log_to_syslog": "false", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "max_rotating_auth_attempts": "10", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_action_on_write_error": "1", 2024-03-24T11:20:35.171 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_alternate_name_max": "8192", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_asio_thread_count": "2", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_export_pin": "true", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_fragment_dirs": "true", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_fragment_interval": "5", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_interval": "10", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_max": "-1", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_max_until": "-1", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_merge_size": "50", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_midchunk": "0.300000", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-24T11:20:35.172 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_min_start": "0.200000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_minchunk": "0.001000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_mode": "0", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_need_max": "1.200000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_need_min": "0.800000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_overload_epochs": "2", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_split_bits": "3", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_split_size": "10000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_target_decay": "10.000000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-24T11:20:35.173 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_beacon_grace": "15.000000", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_beacon_interval": "4.000000", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache_mid": "0.700000", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache_release_free_interval": "10", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache_reservation": "0.050000", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache_trim_interval": "1", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache_trim_threshold": "262144", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-24T11:20:35.174 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_client_prealloc_inos": "1000", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_connect_bootstrapping": "false", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_damage_table_max_entries": "10000", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_debug_auth_pins": "false", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_debug_frag": "false", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_debug_scatterstat": "false", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_debug_subtrees": "false", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_decay_halflife": "5.000000", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_default_dir_hash": "2", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_defer_session_stale": "true", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_deny_all_reconnect": "false", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dir_keys_per_op": "16384", 2024-03-24T11:20:35.175 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dir_max_commit_size": "10", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dir_max_entries": "0", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dir_prefetch": "true", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dump_cache_on_map": "false", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_early_reply": "true", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_enable_op_tracker": "true", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_enforce_unique_name": "true", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_export_ephemeral_random": "true", 2024-03-24T11:20:35.176 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_health_summarize_threshold": "10", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_inject_health_dummy": "false", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-24T11:20:35.177 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_inject_migrator_session_race": "false", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_join_fs": "", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_journal_format": "1", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_create_at": "0", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_export_at": "0", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_import_at": "0", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_journal_at": "0", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_journal_expire_at": "0", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_journal_replay_at": "0", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_link_at": "0", 2024-03-24T11:20:35.178 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_mdstable_at": "0", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_openc_at": "0", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_rename_at": "0", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_shutdown_at": "0", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_event_large_threshold": "524288", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_events_per_segment": "1024", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_max_events": "-1", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_max_segments": "128", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_pause": "false", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_segment_size": "0", 2024-03-24T11:20:35.179 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_trim_threshold": "128", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log_warn_factor": "2.000000", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_caps_per_client": "1048576", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_completed_flushes": "100000", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_completed_requests": "100000", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_export_size": "20971520", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_file_recover": "32", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_purge_files": "64", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_purge_ops": "8192", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-24T11:20:35.180 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_max_snaps_per_dir": "100", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_metrics_update_interval": "2", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_min_caps_per_client": "100", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_min_caps_working_set": "10000", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_numa_node": "-1", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_op_complaint_time": "30.000000", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_op_history_duration": "600", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_op_history_size": "20", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_op_history_slow_op_size": "20", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_op_log_threshold": "5", 2024-03-24T11:20:35.181 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_ping_grace": "15", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_ping_interval": "5", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_recall_max_caps": "30000", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_recall_warning_threshold": "262144", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_replay_interval": "1.000000", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-24T11:20:35.182 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_root_ino_gid": "0", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_root_ino_uid": "0", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_shutdown_check": "0", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_skip_ino": "0", 2024-03-24T11:20:35.183 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_snap_max_uid": "4294967294", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_snap_min_uid": "0", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_snap_rstat": "false", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_standby_replay_damaged": "false", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_symlink_recovery": "true", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_thrash_exports": "0", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_thrash_fragments": "0", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_tick_interval": "5.000000", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_valgrind_exit": "false", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_verify_backtrace": "1", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_verify_scatter": "false", 2024-03-24T11:20:35.184 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_wipe_sessions": "false", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mempool_debug": "false", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "memstore_device_bytes": "1073741824", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "memstore_page_set": "false", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "memstore_page_size": "65536", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_client_bytes": "134217728", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_client_messages": "512", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-24T11:20:35.185 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_disabled_modules": "", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_max_pg_num_change": "128", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_mds_bytes": "134217728", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_mds_messages": "128", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_mon_bytes": "134217728", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_mon_messages": "128", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_osd_bytes": "536870912", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_osd_messages": "8192", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_pool": "true", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_standby_modules": "true", 2024-03-24T11:20:35.186 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_stats_period": "5", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_stats_threshold": "5", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_tick_period": "2", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_allow_pool_delete": "true", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_allow_pool_size_one": "false", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_auth_validate_all_caps": "true", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_bytes": "104857600", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_directed_command_retry": "2", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-24T11:20:35.187 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_hunt_parallel": "3", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_log_interval": "1.000000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_ping_interval": "10.000000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_client_target_rank": "-1", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_file_level": "debug", 2024-03-24T11:20:35.188 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_file": "true", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_journald": "true", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_compact_on_bootstrap": "false", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_compact_on_start": "false", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_compact_on_trim": "true", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-24T11:20:35.189 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_cpu_threads": "4", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_crush_min_required_version": "hammer", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_daemon_bytes": "419430400", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_data_avail_crit": "5", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_data_avail_warn": "5", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_data_size_warn": "16106127360", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_dump_json": "false", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_dump_transactions": "false", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_extra_checks": "false", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-24T11:20:35.190 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_no_require_reef": "false", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_no_require_squid": "false", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_down_mkfs_grace": "60", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_election_default_strategy": "1", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_election_timeout": "5.000000", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_elector_ping_divisor": "2", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_enable_op_tracker": "true", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_fake_pool_delete": "false", 2024-03-24T11:20:35.191 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_force_quorum_join": "false", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_globalid_prealloc": "10000", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_health_detail_to_clog": "true", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_health_log_update_period": "5", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_health_max_detail": "50", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_health_to_clog": "true", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_health_to_clog_interval": "600", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_host": "[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_host_override": "", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_initial_members": "", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-24T11:20:35.192 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_lease": "5.000000", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_log_full_interval": "50", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_log_max": "10000", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_log_max_summary": "50", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_log_epochs": "500", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-24T11:20:35.193 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_osd": "10000", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_pg_per_osd": "10000", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_pool_pg_num": "65536", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mds_force_trim_to": "0", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mds_skip_sanity": "false", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_memory_autotune": "true", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_memory_target": "2147483648", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mgr_beacon_grace": "30", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mgr_digest_period": "5", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mgr_inactive_grace": "60", 2024-03-24T11:20:35.194 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_min_osdmap_epochs": "50", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_op_complaint_time": "30", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_op_history_duration": "600", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_op_history_size": "20", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_op_history_slow_op_size": "20", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_op_log_threshold": "5", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-24T11:20:35.195 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_auto_mark_in": "false", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_cache_size": "500", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_down_out_interval": "600", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_force_trim_to": "0", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-24T11:20:35.196 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_min_down_reporters": "2", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_report_timeout": "900", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-24T11:20:35.197 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pg_stuck_threshold": "60", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-24T11:20:35.198 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_probe_timeout": "2.000000", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_reweight_max_change": "0.050000", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_reweight_max_osds": "4", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_scrub_interval": "86400", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_scrub_max_keys": "100", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_scrub_timeout": "300", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_session_timeout": "300", 2024-03-24T11:20:35.199 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_smart_report_timeout": "5", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_stat_smooth_intervals": "6", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_stretch_pool_min_size": "2", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_stretch_pool_size": "4", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_sync_debug": "false", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_sync_provider_kill_at": "0", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_sync_requester_kill_at": "0", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_sync_timeout": "60.000000", 2024-03-24T11:20:35.200 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_target_pg_per_osd": "100", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_tick_interval": "5", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_timecheck_interval": "300.000000", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_use_min_delay_socket": "false", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_older_version_delay": "604800", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_misplaced": "false", 2024-03-24T11:20:35.201 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_older_version": "true", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "monmap": "", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-24T11:20:35.202 INFO:teuthology.orchestra.run.smithi179.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "motr_tracing_enabled": "false", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_op_threads": "3", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_cm": "false", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_device_name": "", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_dscp": "96", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_local_gid": "", 2024-03-24T11:20:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_port_num": "1", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_sl": "3", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_support_srq": "true", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_rdma_type": "ib", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_async_reap_threshold": "5", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_before_connect": "false", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_ipv4": "true", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_ipv6": "false", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_msgr1": "true", 2024-03-24T11:20:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_msgr2": "true", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_port_max": "7568", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_port_min": "6800", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_retry_count": "3", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_bind_retry_delay": "5", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_blackhole_client": "false", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_blackhole_mds": "false", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_blackhole_mgr": "false", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_blackhole_mon": "false", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_blackhole_osd": "false", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_client_mode": "crc secure", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_cluster_mode": "crc secure", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_cluster_type": "", 2024-03-24T11:20:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_compress_secure": "false", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_connection_idle_timeout": "900", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_connection_ready_timeout": "10", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_crc_data": "true", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_crc_header": "true", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_die_on_bad_msg": "false", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_die_on_bug": "true", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_die_on_old_message": "true", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_die_on_skipped_message": "false", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_coremask": "0xF", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-24T11:20:35.206 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_enable_tso": "true", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_hugepages": "", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_lro": "true", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_memory_channel": "4", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_pmd": "", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_port_id": "0", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-24T11:20:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_dump_on_send": "false", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_initial_backoff": "0.200000", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_inject_delay_max": "1.000000", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_inject_delay_type": "", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_inject_network_congestion": "0", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_inject_socket_failures": "0", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_learn_addr_from_peer": "true", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_max_accept_failures": "4", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_max_backoff": "15.000000", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_mon_client_mode": "secure crc", 2024-03-24T11:20:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_mon_service_mode": "secure crc", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_osd_compress_min_size": "1024", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_osd_compress_mode": "none", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_pq_min_cost": "65536", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_public_type": "", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_service_mode": "crc secure", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_tcp_listen_backlog": "512", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_tcp_nodelay": "true", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_tcp_rcvbuf": "0", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "ms_type": "async+posix", 2024-03-24T11:20:35.209 INFO:teuthology.orchestra.run.smithi179.stdout: "no_config_file": "false", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_completion_locks_per_session": "32", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_inflight_ops": "1024", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_tick_interval": "5.000000", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter_timeout": "10.000000", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objectstore_blackhole": "false", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "openssl_engine_opts": "", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_agent_delay_time": "5.000000", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_agent_hist_halflife": "1000", 2024-03-24T11:20:35.210 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_agent_max_low_ops": "2", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_agent_max_ops": "4", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_agent_slop": "0.020000", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_asio_thread_count": "2", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_async_recovery_min_cost": "100", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_backfill_scan_max": "512", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_backfill_scan_min": "64", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_backoff_on_degraded": "false", 2024-03-24T11:20:35.211 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_backoff_on_peering": "false", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_backoff_on_unfound": "true", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_beacon_report_interval": "300", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bench_duration": "30", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bench_max_block_size": "67108864", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_blkin_trace_all": "false", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_check_for_log_corruption": "false", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-24T11:20:35.212 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_class_update_on_start": "true", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_client_message_cap": "256", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_client_message_size_cap": "524288000", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_client_op_priority": "63", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_client_watch_timeout": "30", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_command_max_records": "256", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_compact_on_start": "false", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_crush_update_on_start": "true", 2024-03-24T11:20:35.213 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_crush_update_weight_set": "true", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_feed_pullee": "-1", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_misdirected_ops": "false", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_no_acting_change": "false", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_no_purge_strays": "false", 2024-03-24T11:20:35.214 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_op_order": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_shutdown": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_trim_objects": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_verify_snaps": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-24T11:20:35.215 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_deep_scrub_keys": "1024", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_deep_scrub_stride": "524288", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_default_notify_timeout": "30", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_delete_sleep": "0.000000", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-24T11:20:35.216 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_discard_disconnected_ops": "true", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_enable_op_tracker": "true", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_fast_info": "true", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_fast_shutdown": "true", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-24T11:20:35.217 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_function_tracing": "false", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_heartbeat_grace": "20", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_heartbeat_interval": "6", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_heartbeat_min_peers": "10", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_heartbeat_min_size": "2000", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_heartbeat_stale": "600", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_hit_set_max_size": "100000", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_hit_set_min_size": "1000", 2024-03-24T11:20:35.218 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_journal_size": "5120", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_kill_backfill_at": "0", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_map_cache_size": "50", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_map_dedup": "true", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_map_message_max": "40", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-24T11:20:35.219 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_map_share_max_epochs": "40", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_attr_name_len": "100", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_attr_size": "0", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_backfills": "1", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_markdown_count": "5", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_markdown_period": "600", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_object_name_len": "2048", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_object_namespace_len": "256", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_object_size": "134217728", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_pg_blocked_by": "16", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_pg_log_entries": "10000", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-24T11:20:35.220 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_pgls": "1024", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_push_cost": "8388608", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_push_objects": "10", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_scrubs": "3", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_trimming_pgs": "2", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_write_op_reply_len": "64", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_max_write_size": "90", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-24T11:20:35.221 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_profile": "balanced", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-24T11:20:35.222 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_memory_base": "805306368", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_memory_cache_min": "134217728", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_memory_target": "4294967296", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_memory_target_autotune": "false", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_min_pg_log_entries": "250", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_min_recovery_priority": "0", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-24T11:20:35.223 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mon_report_interval": "5", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_num_cache_shards": "32", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_num_op_tracker_shard": "32", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_numa_auto_affinity": "true", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_numa_node": "-1", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_numa_prefer_iface": "true", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_objecter_finishers": "1", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_objectstore": "bluestore", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_objectstore_fuse": "false", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_objectstore_tracing": "false", 2024-03-24T11:20:35.224 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_complaint_time": "30.000000", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_history_duration": "600", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_history_size": "20", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_history_slow_op_size": "20", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_log_threshold": "5", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_num_shards": "0", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_num_shards_hdd": "5", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_num_shards_ssd": "8", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-24T11:20:35.225 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_pq_min_cost": "65536", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_queue_cut_off": "high", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_op_thread_timeout": "15", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_open_classes_on_start": "true", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_os_flags": "0", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_peering_op_priority": "255", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_delete_cost": "1048576", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_delete_priority": "5", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_log_trim_max": "10000", 2024-03-24T11:20:35.226 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_log_trim_min": "100", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_crimson": "false", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-24T11:20:35.227 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_flags": "0", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_min_size": "0", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_pg_num": "32", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_pgp_num": "0", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-24T11:20:35.228 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_read_ratio": "70", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_size": "3", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_default_type": "replicated", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_push_per_object_cost": "1000", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recover_clone_overlap": "true", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_cost": "20971520", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_max_active": "0", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-24T11:20:35.229 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_max_single_start": "1", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_op_priority": "3", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_priority": "5", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_sleep": "0.000000", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_repair_during_recovery": "false", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_requested_scrub_priority": "120", 2024-03-24T11:20:35.230 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_auto_repair": "false", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_begin_hour": "0", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_begin_week_day": "0", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_chunk_max": "25", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_chunk_min": "5", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_cost": "52428800", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_during_recovery": "false", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_end_hour": "0", 2024-03-24T11:20:35.231 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_end_week_day": "0", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_event_cost": "4096", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_invalid_stats": "true", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_max_preemptions": "5", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_priority": "5", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-24T11:20:35.232 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_sleep": "0.000000", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_skip_data_digest": "false", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_smart_report_timeout": "5", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_snap_trim_cost": "1048576", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_snap_trim_priority": "5", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-24T11:20:35.233 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_target_transaction_size": "30", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-24T11:20:35.234 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_tracing": "false", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_use_stale_snap": "false", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "osdc_blkin_trace_all": "false", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_kill_at": "0", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_max_join_drift": "10", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_min": "500", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_min_wait": "0.050000", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_propose_interval": "1.000000", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_service_trim_max": "500", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_service_trim_min": "10", 2024-03-24T11:20:35.235 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_stash_full_interval": "25", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_trim_max": "500", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "paxos_trim_min": "250", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "perf": "true", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "pid_file": "", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "-", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrv": "", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "public_bind_addr": "-", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "public_network": "", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "public_network_interface": "", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "qat_compressor_enabled": "false", 2024-03-24T11:20:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: "qat_compressor_session_max_number": "256", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rados_mon_op_timeout": "0", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rados_osd_op_timeout": "0", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rados_tracing": "false", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_atime_update_interval": "60", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_balance_parent_reads": "false", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_balance_snap_reads": "false", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_blkin_trace_all": "false", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache": "true", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-24T11:20:35.237 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_policy": "writearound", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_size": "33554432", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_clone_copy_on_read": "false", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_compression_hint": "none", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_concurrent_management_ops": "10", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_clone_format": "auto", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_data_pool": "", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_features": "61", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_format": "2", 2024-03-24T11:20:35.238 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_map_options": "", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_order": "22", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_pool": "rbd", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_stripe_count": "0", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_default_stripe_unit": "0", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_enable_alloc_hint": "true", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_io_scheduler": "simple", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-24T11:20:35.239 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_order": "24", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_pool": "", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_journal_splay_width": "4", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_localize_parent_reads": "false", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_localize_snap_reads": "false", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-24T11:20:35.240 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-24T11:20:35.241 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-24T11:20:35.242 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_mtime_update_interval": "60", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_non_blocking_aio": "true", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_op_thread_timeout": "60", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_op_threads": "1", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_parent_cache_enabled": "false", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_plugins": "", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_bps_burst": "0", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_bps_limit": "0", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_exclude_ops": "0", 2024-03-24T11:20:35.243 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_iops_burst": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_iops_limit": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-24T11:20:35.244 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_read_from_replica_policy": "default", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_skip_partial_discard": "true", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_tracing": "false", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_validate_names": "true", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd_validate_pool": "true", 2024-03-24T11:20:35.245 INFO:teuthology.orchestra.run.smithi179.stdout: "restapi_base_url": "", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "restapi_log_level": "", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_acl_grants_max_num": "100", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_admin_entry": "admin", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_backend_store": "rados", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_barbican_url": "", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_beast_enable_async": "true", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_counters_cache": "false", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-24T11:20:35.246 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_cache_enabled": "true", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_cache_expiry_interval": "900", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_cache_lru_size": "10000", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_config_store": "rados", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_content_length_compat": "false", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_copy_obj_progress": "true", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_cors_rules_max_num": "100", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_cross_domain_policy": "", 2024-03-24T11:20:35.247 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_addr": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_password": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_kmip_username": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_require_ssl": "true", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-24T11:20:35.248 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_addr": "", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_auth": "token", 2024-03-24T11:20:35.249 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_namespace": "", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_prefix": "", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_token_file": "", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_curl_buffersize": "524288", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_curl_low_speed_time": "300", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-24T11:20:35.250 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_d4n_port": "6379", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data_log_changes_size": "1000", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data_log_num_shards": "128", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-24T11:20:35.251 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data_log_window": "30", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-24T11:20:35.252 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dns_name": "", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dns_s3website_name": "", 2024-03-24T11:20:35.253 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_dynamic_resharding": "true", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enable_gc_threads": "true", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enable_lc_threads": "true", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enable_ops_log": "false", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enable_quota_threads": "true", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enable_static_website": "false", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enable_usage_log": "false", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_enforce_swift_acls": "true", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_exit_timeout_secs": "120", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_expose_bucket": "false", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_extended_http_attrs": "", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_filter": "none", 2024-03-24T11:20:35.254 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_frontends": "beast port=7480", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_max_deferred": "50", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_max_objs": "32", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_gc_processor_period": "3600", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-24T11:20:35.255 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_init_timeout": "300", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_kafka_connection_idle": "30", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_domain": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_password": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_password_path": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_project": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_tenant": "", 2024-03-24T11:20:35.256 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_token": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_token_path": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_admin_user": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_api_version": "2", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_barbican_domain": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_barbican_password": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_barbican_project": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_barbican_user": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_url": "", 2024-03-24T11:20:35.257 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lc_debug_interval": "-1", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lc_lock_max_time": "90", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lc_max_objs": "32", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lc_max_rules": "1000", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lc_max_worker": "3", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lc_thread_delay": "0", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ldap_dnattr": "uid", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ldap_searchfilter": "", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-24T11:20:35.258 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_log_http_headers": "", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_log_object_name_utc": "false", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_attr_name_len": "0", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_attr_size": "0", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_chunk_size": "4194304", 2024-03-24T11:20:35.259 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_listing_results": "1000", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_notify_retries": "10", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_put_param_size": "1048576", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_put_size": "5368709120", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_max_slo_entries": "1000", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_md_log_max_shards": "64", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-24T11:20:35.260 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_mp_lock_max_time": "600", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_max_gc": "300", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-24T11:20:35.261 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_num_async_rados_threads": "32", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_num_control_oids": "8", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_numa_node": "-1", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_objexp_chunk_size": "100", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_objexp_gc_interval": "600", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-24T11:20:35.262 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_op_thread_timeout": "600", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_op_tracing": "false", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_opa_token": "", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_opa_url": "", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_opa_verify_ssl": "true", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ops_log_rados": "false", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_ops_log_socket_path": "", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-24T11:20:35.263 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_period_push_interval": "2.000000", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_posix_cache_lanes": "3", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_posix_cache_partitions": "3", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_print_continue": "true", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-24T11:20:35.264 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_rados_tracing": "false", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_realm": "", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_realm_id": "", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_region": "", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-24T11:20:35.265 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_request_uri": "", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_reshard_batch_size": "64", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_reshard_max_aio": "128", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_reshard_num_logs": "16", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_reshard_thread_interval": "600", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_resolve_cname": "false", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_run_sync_thread": "true", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-24T11:20:35.266 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_scheduler_type": "throttler", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_script_uri": "", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_service_provider_name": "", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sts_client_id": "", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sts_client_secret": "", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sts_entry": "sts", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sts_key": "sts", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-24T11:20:35.267 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sts_min_session_duration": "900", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sts_token_introspection_url": "", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_account_in_url": "false", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_auth_entry": "auth", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_auth_url": "", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_custom_header": "", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_need_stats": "true", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_tenant_name": "", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_token_expiration": "86400", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_url": "", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_url_prefix": "swift", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-24T11:20:35.268 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_lease_period": "120", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_thread_pool_size": "512", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-24T11:20:35.269 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_comment": "", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_createby": "", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_encoding": "", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_flag": "false", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_origin": "", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_torrent_tracker": "", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_trust_forwarded_https": "false", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_usage_max_shards": "32", 2024-03-24T11:20:35.270 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_usage_max_user_shards": "1", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_use_opa_authz": "false", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_counters_cache": "false", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_max_buckets": "1000", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_policies_max_num": "100", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_user_unique_email": "true", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_verify_ssl": "true", 2024-03-24T11:20:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_zone": "", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_zone_id": "", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_zonegroup": "", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_zonegroup_id": "", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgwlc_auto_session_clear": "true", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_block_size": "4096", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-24T11:20:35.272 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cache_size": "536870912", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_index_type": "binary_search", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-24T11:20:35.273 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_partition_filters": "false", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_perf": "false", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "run_dir": "/var/run/ceph", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_block_create": "true", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_cache_lru_size": "67108864", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_default_max_object_size": "16777216", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_device_size": "53687091200", 2024-03-24T11:20:35.274 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_journal_batch_capacity": "16", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_main_device_type": "SSD", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "seastore_segment_size": "67108864", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "set_keepcaps": "false", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "setgroup": "ceph", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "setuser": "ceph", 2024-03-24T11:20:35.275 INFO:teuthology.orchestra.run.smithi179.stdout: "setuser_match_path": "", 2024-03-24T11:20:35.276 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-24T11:20:35.276 INFO:teuthology.orchestra.run.smithi179.stdout: "thp": "false", 2024-03-24T11:20:35.276 INFO:teuthology.orchestra.run.smithi179.stdout: "threadpool_default_timeout": "60", 2024-03-24T11:20:35.276 INFO:teuthology.orchestra.run.smithi179.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-24T11:20:35.276 INFO:teuthology.orchestra.run.smithi179.stdout: "throttler_perf_counter": "true" 2024-03-24T11:20:35.276 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:20:35.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:35 smithi181 ceph-mon[33998]: 3.e deep-scrub starts 2024-03-24T11:20:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:35 smithi181 ceph-mon[33998]: 3.e deep-scrub ok 2024-03-24T11:20:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:35 smithi181 ceph-mon[33998]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:35 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:20:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:35 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:20:35.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:20:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:20:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[29608]: 3.e deep-scrub starts 2024-03-24T11:20:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[29608]: 3.e deep-scrub ok 2024-03-24T11:20:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[29608]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:20:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:20:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[37907]: 3.e deep-scrub starts 2024-03-24T11:20:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[37907]: 3.e deep-scrub ok 2024-03-24T11:20:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[37907]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:20:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:35 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:20:35.959 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:35 smithi179 podman[73895]: 2024-03-24 11:20:35.523067176 +0000 UTC m=+3.321934925 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:20:35.959 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:35 smithi179 podman[73895]: 2024-03-24 11:20:35.771637376 +0000 UTC m=+3.570505125 container create e4eaeeb17d51cff887b431ded96f15859e9081c9822e812e397380935e85f2ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:20:36.275 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:20:36.275 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:20:36.275 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:20:36.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 podman[73895]: 2024-03-24 11:20:36.141558066 +0000 UTC m=+3.940425835 container init e4eaeeb17d51cff887b431ded96f15859e9081c9822e812e397380935e85f2ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 podman[73895]: 2024-03-24 11:20:36.144236759 +0000 UTC m=+3.943104516 container start e4eaeeb17d51cff887b431ded96f15859e9081c9822e812e397380935e85f2ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.148Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.148Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.148Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.148Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:20:36.276 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:20:36.277 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:20:36.277 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:20:36.277 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:20:36.277 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:20:36.277 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:20:36.277 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:20:36.277 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:20:36.278 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:20:36.279 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.149Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a[74061]: ts=2024-03-24T11:20:36.150Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:20:36.280 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 bash[73895]: e4eaeeb17d51cff887b431ded96f15859e9081c9822e812e397380935e85f2ca 2024-03-24T11:20:36.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:36 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:20:36.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:36 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:20:36.709 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:20:36 smithi179 systemd[1]: Started Ceph node-exporter.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:20:36.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-24T11:20:37.092 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: 3.1e scrub starts 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: 3.1e scrub ok 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[29608]: Deploying daemon node-exporter.b on smithi181 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: 3.1e scrub starts 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: 3.1e scrub ok 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:37 smithi179 ceph-mon[37907]: Deploying daemon node-exporter.b on smithi181 2024-03-24T11:20:37.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: 3.1e scrub starts 2024-03-24T11:20:37.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: 3.1e scrub ok 2024-03-24T11:20:37.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:37.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:20:37.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:20:37.649 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.650 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.650 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:37.650 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:37 smithi181 ceph-mon[33998]: Deploying daemon node-exporter.b on smithi181 2024-03-24T11:20:38.654 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:38 smithi181 bash[65071]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T11:20:39.049 INFO:teuthology.orchestra.run.smithi179.stdout:mon.a: {} 2024-03-24T11:20:39.060 INFO:teuthology.orchestra.run.smithi179.stdout:mon.b: {} 2024-03-24T11:20:39.070 INFO:teuthology.orchestra.run.smithi179.stdout:mon.c: {} 2024-03-24T11:20:39.071 INFO:teuthology.orchestra.run.smithi179.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:20:39.071 INFO:teuthology.orchestra.run.smithi179.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:20:39.071 INFO:teuthology.orchestra.run.smithi179.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:20:39.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:38 smithi181 bash[65071]: Getting image source signatures 2024-03-24T11:20:39.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:38 smithi181 bash[65071]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T11:20:39.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:38 smithi181 bash[65071]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T11:20:39.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:38 smithi181 bash[65071]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T11:20:39.620 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[29608]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:39.621 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:20:39.621 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:20:39.621 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3423819129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:20:39.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[37907]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:39.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:20:39.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:20:39.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:39 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3423819129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:20:39.648 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-24T11:20:39.659 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T11:20:39.670 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T11:20:39.670 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-24T11:20:39.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-24T11:20:39.671 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-24T11:20:39.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:20:39.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:20:39.671 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-24T11:20:39.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-24T11:20:39.672 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-24T11:20:39.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:39.672 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-24T11:20:39.673 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:20:39.673 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-24T11:20:39.673 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:20:39.674 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-24T11:20:39.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:20:39.675 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:39 smithi181 ceph-mon[33998]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:39.676 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:39 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:20:39.676 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:39 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:20:39.676 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:39 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3423819129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:20:39.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:39.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:39.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:39.835 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:39.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:39.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:39.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:39.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:39.979 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:39.982 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:39.982 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:40.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.095 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:40.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:40.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:40.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:40.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:40.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:40.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:40.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:40 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:20:40.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:40 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:20:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:40 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:20:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:40 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:20:40.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:40.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:40.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:40.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:40 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:20:40.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:40 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:20:40.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.742 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:40.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:40.843 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.868 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:40.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:40.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:40.997 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:40.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:41.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:41.126 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:41.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:41.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:41.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:41.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:41.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:41.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:41.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:41.448 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:20:41.448 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:20:41.448 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:20:41.449 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:20:41.450 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:20:41.450 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:20:41.450 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:20:41.450 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:41 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:20:41.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:41.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:41.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:41.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:41.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:41.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:20:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:41 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:20:41.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:41.764 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:41.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:41.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:41.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:41.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:41.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:42.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:42.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.154 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:42.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:42.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:42.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:42.385 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:42.401 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[29608]: 6.c scrub starts 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[29608]: 6.c scrub ok 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[29608]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[29608]: 4.14 scrub starts 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[29608]: 4.14 scrub ok 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[37907]: 6.c scrub starts 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[37907]: 6.c scrub ok 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:20:42.402 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:20:42.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[37907]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:20:42.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[37907]: 4.14 scrub starts 2024-03-24T11:20:42.403 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:42 smithi179 ceph-mon[37907]: 4.14 scrub ok 2024-03-24T11:20:42.410 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:42.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:42.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:42.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:42.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:42 smithi181 ceph-mon[33998]: 6.c scrub starts 2024-03-24T11:20:42.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:42 smithi181 ceph-mon[33998]: 6.c scrub ok 2024-03-24T11:20:42.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:42 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:20:42.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:42 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:20:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:42 smithi181 ceph-mon[33998]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T11:20:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:42 smithi181 ceph-mon[33998]: 4.14 scrub starts 2024-03-24T11:20:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:42 smithi181 ceph-mon[33998]: 4.14 scrub ok 2024-03-24T11:20:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:42.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:42.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:42.797 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:42.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:42.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:42.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:43.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:43.140 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:43.140 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:43.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:43.158 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:20:43.158 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:20:43.158 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 0 2024-03-24T11:20:43.158 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:20:43.182 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:43.284 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.308 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:43.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:20:43.409 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:43.433 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[29608]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[29608]: 4.11 scrub starts 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[29608]: 4.11 scrub ok 2024-03-24T11:20:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:20:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:20:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[37907]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:20:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:20:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[37907]: 4.11 scrub starts 2024-03-24T11:20:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:43 smithi179 ceph-mon[37907]: 4.11 scrub ok 2024-03-24T11:20:43.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.560 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:43.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:20:43.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:20:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-mon[33998]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:20:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:20:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-mon[33998]: 4.11 scrub starts 2024-03-24T11:20:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:43 smithi181 ceph-mon[33998]: 4.11 scrub ok 2024-03-24T11:20:43.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:43.685 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:43.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:43.810 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:43.912 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:43.937 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:43.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:44.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:20:44.088 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:20:44.089 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:20:44.089 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:20:44.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.190 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:44.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:44.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.316 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:44.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:44.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:44.441 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:44.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.567 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:44.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:44.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:44 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:20:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:44 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:20:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:44 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:20:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:44 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:20:44.668 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:44.692 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:20:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:44 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:20:44.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:44.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:20:44.840 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:20:44.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-24T11:20:44.919 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:44.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:44.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:45.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.070 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:45.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:45.151 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:44 smithi181 bash[65071]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:20:45.151 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 bash[65071]: Writing manifest to image destination 2024-03-24T11:20:45.158 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:20:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:20:45.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:45.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:45.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.321 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:45.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:45.422 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:45.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:45.457 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:45 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:20:45.457 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:45 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:20:45.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:45 smithi179 ceph-mon[29608]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:45.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:45 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:20:45.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:45 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:20:45.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:45 smithi179 ceph-mon[37907]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:45.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:20:45.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:45.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:45.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:45.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:20:45.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:20:45.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-mon[33998]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:45.628 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 podman[65071]: 2024-03-24 11:20:45.268605456 +0000 UTC m=+6.956380393 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:20:45.628 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 podman[65071]: 2024-03-24 11:20:45.3670315 +0000 UTC m=+7.054806428 container create 1f5d71af48a9896e4e9b21c1def1010da9650bbe66c8696a56a8cafb4c54389c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:20:45.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:45.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:45.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:45.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.826 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:45.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:45.902 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 podman[65071]: 2024-03-24 11:20:45.627891333 +0000 UTC m=+7.315666267 container init 1f5d71af48a9896e4e9b21c1def1010da9650bbe66c8696a56a8cafb4c54389c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:20:45.902 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 podman[65071]: 2024-03-24 11:20:45.63061252 +0000 UTC m=+7.318387452 container start 1f5d71af48a9896e4e9b21c1def1010da9650bbe66c8696a56a8cafb4c54389c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:20:45.902 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.634Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:20:45.902 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.634Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:20:45.902 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.634Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.634Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.634Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.634Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:20:45.903 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:20:45.904 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:20:45.905 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:20:45.906 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b[65512]: ts=2024-03-24T11:20:45.635Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 bash[65071]: 1f5d71af48a9896e4e9b21c1def1010da9650bbe66c8696a56a8cafb4c54389c 2024-03-24T11:20:45.907 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:20:45 smithi181 systemd[1]: Started Ceph node-exporter.b for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:20:45.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:45.950 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:45.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:46.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:46.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:46.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:46.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:46.304 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.329 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:46.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:46.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:46.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:46.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:46.583 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:46.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:46.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:46.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:46.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:46.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: 6.1f scrub ok 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:46.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:20:46.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:20:46.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:20:46.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:20:46.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:20:46.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: 6.1f scrub ok 2024-03-24T11:20:46.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:20:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:46 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:20:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:46.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:47.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:47.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: 6.1f scrub ok 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:47.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:47.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:47.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:20:47.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:46 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:20:47.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:47.214 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:47.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:47.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:47.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:47.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:47.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:47.591 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:47.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.737 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:47.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:47.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:47.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:47.891 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:47.892 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:47.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:47.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:47.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:48.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[29608]: 4.17 scrub starts 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[29608]: 4.17 scrub ok 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[37907]: 4.17 scrub starts 2024-03-24T11:20:48.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[37907]: 4.17 scrub ok 2024-03-24T11:20:48.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:48.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:48.113 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:48.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:47 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:20:48.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:47 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:20:48.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:47 smithi181 ceph-mon[33998]: 4.17 scrub starts 2024-03-24T11:20:48.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:47 smithi181 ceph-mon[33998]: 4.17 scrub ok 2024-03-24T11:20:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:20:48.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.240 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:48.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:48.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:48.365 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:48.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:48.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:48.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:48.616 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:48.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:48.742 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:48.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.869 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:48.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:48.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:48.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:48.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:49.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:49.105 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:48 smithi179 ceph-mon[29608]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:49.106 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:48 smithi179 ceph-mon[37907]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:49.119 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:49.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:49.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:48 smithi181 ceph-mon[33998]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:49.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:49.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:49.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:49.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:49.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:49.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:49.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:49.496 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:49.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:49.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:49.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:49.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:49.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:49.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:49.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:49.848 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:49.873 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:49.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:49.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:49 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:20:49.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:49 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:20:49.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:49 smithi181 ceph-mon[33998]: 6.2 deep-scrub starts 2024-03-24T11:20:49.964 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:49 smithi181 ceph-mon[33998]: 6.2 deep-scrub ok 2024-03-24T11:20:49.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:49.999 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:20:50.000 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:50.012 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[29608]: 6.2 deep-scrub starts 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[29608]: 6.2 deep-scrub ok 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[37907]: 6.2 deep-scrub starts 2024-03-24T11:20:50.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:49 smithi179 ceph-mon[37907]: 6.2 deep-scrub ok 2024-03-24T11:20:50.032 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:20:50.051 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:20:50.070 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:20:50.090 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:20:50.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:50.110 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:20:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:50.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:50.130 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:20:50.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:20:50.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.253 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:50.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:50.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:50.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:50.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:50.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:50.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:50.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:50.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:50.756 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:50.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:50.882 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:50.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:50.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.009 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:51.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:51.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:51.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:51.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1493341354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2331171174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:20:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/785940818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:20:51.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:50 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:20:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:20:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:20:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:20:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:20:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:20:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1493341354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2331171174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/785940818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:20:51.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1493341354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2331171174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/785940818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:50 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:20:51.232 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_0' created 2024-03-24T11:20:51.232 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:20:51.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:51.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:51.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:51.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:51.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.511 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:51.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:51.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:51.637 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:51.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:51.738 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:51.762 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:51.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:51.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:20:51.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T11:20:51.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:20:51.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[29608]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:20:51.888 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:51.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:51.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:52.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:52.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:52.138 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:20:52.138 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T11:20:52.138 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:20:52.138 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:51 smithi179 ceph-mon[37907]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:20:52.139 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:52.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:51 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:20:52.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:51 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T11:20:52.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:51 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:20:52.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:51 smithi181 ceph-mon[33998]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T11:20:52.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.268 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:52.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:52.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:52.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:52.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:52.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:52.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:52.645 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:52.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:52.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:52.770 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:52.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:52.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:52.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:52.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:53.022 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:53.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.151 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:53.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:53.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:53.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:53.353 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:53 smithi179 ceph-mon[29608]: pgmap v52: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:20:53.353 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:53 smithi179 ceph-mon[29608]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:20:53.353 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:53 smithi179 ceph-mon[29608]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:53.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:53 smithi179 ceph-mon[37907]: pgmap v52: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:20:53.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:53 smithi179 ceph-mon[37907]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:20:53.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:53 smithi179 ceph-mon[37907]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:53.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:53.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:53 smithi181 ceph-mon[33998]: pgmap v52: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:20:53.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:53 smithi181 ceph-mon[33998]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T11:20:53.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:53 smithi181 ceph-mon[33998]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:53.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:20:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:20:53.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:53.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:53.529 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:53.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:53.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:53.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.785 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:53.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:53.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:53.915 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:53.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:54.017 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.043 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:54.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:54.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:54.154 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:20:54.154 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:20:54.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:54.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:54.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:54.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.430 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:54.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:54.531 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:54.553 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:54.554 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:54 smithi179 ceph-mon[29608]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:20:54.554 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:54.554 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:54 smithi179 ceph-mon[37907]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:20:54.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:54.579 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-24T11:20:54.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:20:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:54 smithi181 ceph-mon[33998]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T11:20:54.657 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:54.686 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:54.788 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.813 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:54.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:54.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:54.940 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:54.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:55.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:55.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:55.172 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:55.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:55.292 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:55 smithi179 ceph-mon[29608]: pgmap v55: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:55.292 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:55 smithi179 ceph-mon[37907]: pgmap v55: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:55.293 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:20:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:20:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:20:55.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:55.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:20:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.324 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:55.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:55.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.451 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:55.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:55.552 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.577 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:55.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:55 smithi181 ceph-mon[33998]: pgmap v55: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:20:55.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:55.705 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:20:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:20:55.733 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:20:55.733 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:20:55.733 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:20:55.733 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:20:55.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:55.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:55.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:55.963 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:55.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:56.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:56.089 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:56.155 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:20:56.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:56.221 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:56.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:56.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:56.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.473 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:56.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:56.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:56.598 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:56.700 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:56.725 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:56.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:56.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:56.958 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:56.982 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:56.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:57.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:57.111 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:57.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:20:57.148 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:20:57.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-24T11:20:57.172 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:57.172 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:57.177 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:57.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:57.240 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:57.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:57.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:57.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:57.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:57.425 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:57 smithi179 ceph-mon[29608]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-24T11:20:57.425 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:57 smithi179 ceph-mon[37907]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-24T11:20:57.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:57.495 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:57.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:57.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:57.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:57.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:57.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:57 smithi181 ceph-mon[33998]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-24T11:20:57.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:57.753 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:57.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:57.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:57.882 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:57.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:57.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:20:58.012 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:20:58.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:20:58.137 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:20:58.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.263 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:58.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:20:58.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:58.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:58.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:20:58.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:58.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:58.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:58.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:58.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:58.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:58.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:58.908 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:58.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:59.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:20:59.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:20:59.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:59.167 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:59.269 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:20:59.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:20:59.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:59.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:59.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:20:59.552 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:20:59.568 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:20:59 smithi179 ceph-mon[29608]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-03-24T11:20:59.568 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:20:59 smithi179 ceph-mon[37907]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-03-24T11:20:59.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:20:59 smithi181 ceph-mon[33998]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-03-24T11:20:59.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:20:59.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:20:59.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:20:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:20:59.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:20:59.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:20:59.937 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:20:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:20:59.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:00.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:00.062 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:00.164 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:00.188 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:00.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:00.318 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:00.420 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:00.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:00.447 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:21:00.447 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:21:00.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:00.569 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:00.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:00.694 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:00.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:00.824 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:00.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:00.954 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:00.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:01.056 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:01.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:01.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:01.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:01.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:01.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:01.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:01.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:01.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:01.585 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:01 smithi179 ceph-mon[29608]: pgmap v58: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:01.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:01 smithi179 ceph-mon[37907]: pgmap v58: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:01.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:01.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:01 smithi181 ceph-mon[33998]: pgmap v58: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:01.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.720 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:01.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:01.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:01.845 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:01.947 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:01.971 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:01.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:02.072 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:02.099 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:02.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:02.224 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:02.267 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:02.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:02.350 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:02.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.477 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:02.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:02.540 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:21:02.540 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:21:02.540 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[29608]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:21:02.540 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:02.541 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:21:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:21:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[37907]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:21:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:02 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:02.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:02.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:02.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:02 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:21:02.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:02 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:21:02.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:02 smithi181 ceph-mon[33998]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T11:21:02.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:02.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:02 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:02.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:02.729 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:02.761 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:21:02.780 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:21:02.799 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:21:02.818 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:21:02.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:02.839 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:21:02.856 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:02.861 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:21:02.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:21:02.901 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:21:02.902 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:21:02.902 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:21:02.902 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:21:02.902 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:21:02.903 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:21:02.903 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:21:02.903 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:21:02.903 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:21:02.957 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:02.980 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:02.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:03.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:03.106 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:03.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:03.231 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:03.269 INFO:teuthology.orchestra.run.smithi179.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-24T11:21:03.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:03.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:03 smithi181 ceph-mon[33998]: pgmap v60: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-24T11:21:03.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:03 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3004504238' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:21:03.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:03 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:21:03.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:03 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3674082928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:03.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:21:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:21:03.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:03.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:03.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:03.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:03.629 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[29608]: pgmap v60: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-24T11:21:03.629 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3004504238' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:21:03.630 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:21:03.630 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3674082928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:03.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[37907]: pgmap v60: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-24T11:21:03.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3004504238' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:21:03.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:21:03.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3674082928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:03.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:03.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:03.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:03.866 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:03.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:03.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:03.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:04.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:04.120 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:04.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:04.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:04.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:04.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:04.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:04.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:04.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:04.378 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:04.378 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:04 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:21:04.379 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:04 smithi179 ceph-mon[29608]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:21:04.379 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:04.379 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:04 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:21:04.379 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:04 smithi179 ceph-mon[37907]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:21:04.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:04.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:04.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:21:04.607 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:21:04.608 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:21:04.608 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:21:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:04.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:04.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:04.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:04.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:04 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:21:04.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:04 smithi181 ceph-mon[33998]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T11:21:04.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:04.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:04.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:04.851 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:04.877 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:04.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:04.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:05.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:05.103 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:05.127 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:05.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:05.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:05.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:05.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:05.383 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:05 smithi179 ceph-mon[29608]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:05.383 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:05 smithi179 ceph-mon[29608]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:21:05.383 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:21:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:05.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:05 smithi179 ceph-mon[37907]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:05.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:05 smithi179 ceph-mon[37907]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:21:05.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:05.504 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:05.606 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:05.631 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:05.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:05 smithi181 ceph-mon[33998]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:05.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:05 smithi181 ceph-mon[33998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:21:05.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:05.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:05.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:05.883 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:05.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:05.986 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:06.050 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:06.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:06.177 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:06.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:06.302 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:06.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:06.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:06.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:06.556 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:06.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:06.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:21:06.735 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:21:06.735 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-24T11:21:06.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.807 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:06.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:06.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:06.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:06.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:06.988 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-24T11:21:06.990 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T11:21:06.991 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T11:21:07.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:07.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:hints = 1 2024-03-24T11:21:07.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T11:21:07.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object prefix: benchmark_data_smithi179_78267 2024-03-24T11:21:07.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:21:07.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-03-24T11:21:07.067 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:07.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:07.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:07.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:07.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:07.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:21:07.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:07.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:07.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:07.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:07.409 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:21:07.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:07.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:07 smithi179 ceph-mon[29608]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:07.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:07 smithi179 ceph-mon[37907]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:07.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:07.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:07.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:07.601 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:07.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:07 smithi181 ceph-mon[33998]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:21:07.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:07.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:07.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:07.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:07.825 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:07.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:07.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:07.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:07.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:08.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 1 15 2984 2969 185.545 185.562 0.00952981 0.00536618 2024-03-24T11:21:08.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:08.141 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:08.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:08.244 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:08.298 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:08.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:08.400 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:08.449 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:08.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:08.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:08.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:08.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:08.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:08.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:08.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:08.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:08.877 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:08.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:08.979 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:09.020 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:09.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 2 16 5747 5731 179.078 172.625 0.00367908 0.00557624 2024-03-24T11:21:09.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:09.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:09.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:21:09.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:21:09.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[29608]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:21:09.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:09.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:09.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[29608]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:21:09.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:21:09.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:21:09.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[37907]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:21:09.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:09.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:09.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:09 smithi179 ceph-mon[37907]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:21:09.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:09.300 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:09.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:09 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:21:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:09 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:21:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:09 smithi181 ceph-mon[33998]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:21:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:09 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:09 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:09 smithi181 ceph-mon[33998]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T11:21:09.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:09.442 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.451 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:09.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:09.473 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:09.481 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:09.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:09.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:09.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:09.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:09.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:09.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:09.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:09.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:09.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:10.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 3 16 8247 8231 171.454 156.25 0.00724453 0.00582049 2024-03-24T11:21:10.097 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:10.141 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:10.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:10.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:10.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:10.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:10.305 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:21:10.305 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:21:10.305 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:10.305 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:10.305 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:10.306 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:21:10.306 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:21:10.306 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:10.306 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:10.306 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:10 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:10.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:10.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:10 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:21:10.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:10 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:21:10.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:10 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:10.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:10 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:10.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:10 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:10.427 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:10.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:10.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:10.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:10.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:10.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:10.713 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:10.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:10.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:10.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:10.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:10.947 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:10.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:10.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:11.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 4 16 10769 10753 167.994 157.625 0.00306207 0.00594013 2024-03-24T11:21:11.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:11.126 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:11.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:11.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[29608]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:21:11.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[29608]: Reconfiguring daemon alertmanager.a on smithi179 2024-03-24T11:21:11.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[29608]: pgmap v66: 148 pgs: 148 active+clean; 179 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 358 op/s 2024-03-24T11:21:11.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[29608]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:21:11.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[37907]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:21:11.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[37907]: Reconfiguring daemon alertmanager.a on smithi179 2024-03-24T11:21:11.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[37907]: pgmap v66: 148 pgs: 148 active+clean; 179 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 358 op/s 2024-03-24T11:21:11.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:11 smithi179 ceph-mon[37907]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:21:11.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:11.271 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:11.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:11.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:11.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:11 smithi181 ceph-mon[33998]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:21:11.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:11 smithi181 ceph-mon[33998]: Reconfiguring daemon alertmanager.a on smithi179 2024-03-24T11:21:11.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:11 smithi181 ceph-mon[33998]: pgmap v66: 148 pgs: 148 active+clean; 179 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 22 MiB/s wr, 358 op/s 2024-03-24T11:21:11.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:11 smithi181 ceph-mon[33998]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T11:21:11.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:11.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:11.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:11.562 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:11.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:11.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:11.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:11.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:11.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:11.844 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:11.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:11.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:11.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:11.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:12.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 5 16 13432 13416 167.68 166.438 0.00549176 0.00595616 2024-03-24T11:21:12.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:12.080 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:21:12.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:12.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:12.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:12.240 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:12.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:12.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:12.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:12.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:12.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:12 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/890879213' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:12.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:12 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/890879213' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:12.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:12 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:12 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/890879213' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:12 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:12.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:12.515 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:12.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:12.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:12.658 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:12.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:12.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:12.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:12.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:12.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:12.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:12.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:13.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:13.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:13.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:13.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 6 16 15658 15642 162.919 139.125 0.00470601 0.00613113 2024-03-24T11:21:13.162 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:13.205 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:13.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:13.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:13.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:13.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:13.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:13 smithi181 ceph-mon[33998]: pgmap v68: 148 pgs: 148 active+clean; 179 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 22 MiB/s wr, 358 op/s 2024-03-24T11:21:13.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:13 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:21:13.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:13 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:21:13.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:13 smithi181 ceph-mon[33998]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:21:13.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:21:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:21:13.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[29608]: pgmap v68: 148 pgs: 148 active+clean; 179 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 22 MiB/s wr, 358 op/s 2024-03-24T11:21:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:21:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:21:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[29608]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:21:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[37907]: pgmap v68: 148 pgs: 148 active+clean; 179 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 22 MiB/s wr, 358 op/s 2024-03-24T11:21:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:21:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:21:13.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:13 smithi179 ceph-mon[37907]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T11:21:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:13.481 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:13.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:13.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:13.612 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:13.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:13.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:13.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:13.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:13.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:13.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:13.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:13.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:14.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:14.062 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 7 16 17896 17880 159.625 139.875 0.00395813 0.00625598 2024-03-24T11:21:14.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:14.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:14.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:14.191 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:14.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:14.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:14.291 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:14.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:14.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:14.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:14.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:14.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:14.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:14.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:14.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:14.833 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:14.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:14.974 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:14.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:15.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 8 16 20276 20260 158.264 148.75 0.00810343 0.00631149 2024-03-24T11:21:15.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:15.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:15.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:15.195 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:15 smithi179 ceph-mon[29608]: pgmap v70: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-03-24T11:21:15.195 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:21:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:15.196 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:15 smithi179 ceph-mon[37907]: pgmap v70: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-03-24T11:21:15.207 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:15.253 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:15.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:15.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:15.402 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:15.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:15.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:15 smithi181 ceph-mon[33998]: pgmap v70: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-03-24T11:21:15.503 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:15.542 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:15.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:15.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:15.683 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:15.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:15.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:15.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:15.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:15.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:15.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:15.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:16.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:16.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 9 16 22223 22207 154.199 121.688 0.010601 0.00647759 2024-03-24T11:21:16.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:16.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:16.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:16.221 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:16.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:16.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:16.356 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:16.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:16.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:16.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:16.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:16.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:16.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:16.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:16.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:16.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:16.753 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:16.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:16.855 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:16.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:16.897 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:16.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:16.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:17.041 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:17.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 10 16 24483 24467 152.903 141.25 0.00783215 0.00653343 2024-03-24T11:21:17.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:17.175 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:17 smithi179 ceph-mon[29608]: pgmap v71: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 663 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:21:17.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:17 smithi179 ceph-mon[37907]: pgmap v71: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 663 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:21:17.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.320 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:17.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:17.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:17 smithi181 ceph-mon[33998]: pgmap v71: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 663 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:21:17.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:17.453 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:17.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:17.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.586 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:17.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:17.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:17.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:17.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.844 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:17.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:17.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:17.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:17.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:18.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 11 16 26367 26351 149.707 117.75 0.00509289 0.00667471 2024-03-24T11:21:18.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:18.121 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:18.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:18.222 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:18.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:18.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:18.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:18.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:18.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:18.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:18.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:18.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:21:18.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:21:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:21:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:18 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:21:18.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:18.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:18.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:18.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:18.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:18 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:18.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:18.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:18 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:21:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:18 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:21:18.652 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:18.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:18.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:18.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:18.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:18.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:18.922 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:18.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:18.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:21:19.024 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:19.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 12 16 27957 27941 145.512 99.375 0.00357649 0.00686675 2024-03-24T11:21:19.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:19.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:19.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:19.228 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:19.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:19.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:19.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:19.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:19.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:19 smithi179 ceph-mon[29608]: pgmap v72: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-24T11:21:19.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:19 smithi179 ceph-mon[29608]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:21:19.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:19 smithi179 ceph-mon[37907]: pgmap v72: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-24T11:21:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:19 smithi179 ceph-mon[37907]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:21:19.463 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:19.504 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:19.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:19.605 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:19.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:19.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:19.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:19 smithi181 ceph-mon[33998]: pgmap v72: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-24T11:21:19.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:19 smithi181 ceph-mon[33998]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T11:21:19.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:19.788 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:19.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:19.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:19.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:19.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:20.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:20.057 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:20.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:20.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 13 16 29599 29583 142.212 102.625 0.0045675 0.00702527 2024-03-24T11:21:20.159 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:20.200 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:20.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:20.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:20.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:20.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:20.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:20.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:20.480 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:20.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:20.581 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:20.617 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:20.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:20.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:20.758 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:20.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:20.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:20.884 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:20.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:20.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:21.026 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:21.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:21.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 14 16 31290 31274 139.603 105.688 0.00224619 0.00715014 2024-03-24T11:21:21.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:21.166 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:21.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:21.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:21.299 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:21.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:21.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:21.443 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:21.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:21.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:21 smithi179 ceph-mon[29608]: pgmap v74: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-03-24T11:21:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:21 smithi179 ceph-mon[37907]: pgmap v74: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-03-24T11:21:21.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:21.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:21.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:21.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:21 smithi181 ceph-mon[33998]: pgmap v74: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 183 MiB/s wr, 2.93k op/s 2024-03-24T11:21:21.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:21.716 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:21.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:21.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:21.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:21.859 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:21.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:21.960 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:21.999 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:22.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 15 16 33096 33080 137.82 112.875 0.00968565 0.00725046 2024-03-24T11:21:22.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:22.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:22.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:22.273 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:22.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:22.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:22.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:22.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:22.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:22.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:22.771 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:22.809 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:22.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:22.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:22.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:23.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:23.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 16 16 35045 35029 136.818 121.812 0.00540902 0.00730188 2024-03-24T11:21:23.085 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:23.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:23.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:23.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:23.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:23.317 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:23.363 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:23.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:23.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:23 smithi181 ceph-mon[33998]: pgmap v75: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-03-24T11:21:23.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:21:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:21:23.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:23 smithi179 ceph-mon[29608]: pgmap v75: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-03-24T11:21:23.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:23 smithi179 ceph-mon[37907]: pgmap v75: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-03-24T11:21:23.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:23.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:23.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:23.598 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:23.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:23.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:23.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:23.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:23.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:23.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:23.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:23.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:24.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:24.043 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:24.063 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 17 16 36859 36843 135.439 113.375 0.00736394 0.00737885 2024-03-24T11:21:24.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:24.178 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:24.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:24.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:24.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:24.447 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:24.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:24.457 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:24.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:24.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:24.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:24.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:24.714 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:21:24.714 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:21:24.715 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:21:24.719 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:24.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:24.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:24.956 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:24.992 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:24.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:25.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 18 15 38559 38544 133.82 106.312 0.00285572 0.00746767 2024-03-24T11:21:25.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:25.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:25.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:25.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:25.266 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:25.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:25.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:25.399 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:25.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:25.406 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:25 smithi179 ceph-mon[29608]: pgmap v76: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:25.406 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:21:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:25.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:25 smithi179 ceph-mon[37907]: pgmap v76: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:25.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:25.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:25.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:25.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:25.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:25 smithi181 ceph-mon[33998]: pgmap v76: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:25.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:25.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:25.709 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:25 smithi179 systemd[1]: Stopping Ceph alertmanager.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:21:25.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:25.827 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:25.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:25.878 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:21:25.920 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:21:25.929 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:25.964 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:21:25.967 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:37619 7.2 e100/92 7:41c888de:::benchmark_data_smithi179_78267_object37618:head v 100'2458, mlcod=100'2454)", 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:24.539126+0000", 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.003447037, 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034835059999999999, 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.539126+0000", 2024-03-24T11:21:25.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.539126+0000", 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.539127+0000", 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5799999999999999e-07 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:25.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.541318+0000", 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0021909329999999999 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.541330+0000", 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2592000000000001e-05 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.541339+0000", 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4449999999999996e-06 2024-03-24T11:21:25.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.541446+0000", 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107732 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.541467+0000", 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0837000000000001e-05 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:25.971 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:21:25.972 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 1 2024-03-24T11:21:25.972 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:21:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:25.973 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:25.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:26.006 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:21:26.044 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:21:26.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 19 16 40227 40211 132.26 104.188 0.00975001 0.00755318 2024-03-24T11:21:26.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:26.085 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:21:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:26.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:26.127 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:21:26.168 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:21:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:21:26.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:26.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:26.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:26.317 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:26 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/668015236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:26.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:26 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/668015236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:26.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:26.383 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:26.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:26.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:26.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:26.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:26.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:26 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/668015236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:26.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:26.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:26.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:26.796 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:26.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:26.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:26.935 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:26.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:27.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:27.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:21:27.064778+0000 min lat: 0.00140901 max lat: 0.160881 avg lat: 0.00761657 2024-03-24T11:21:27.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:21:27.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 20 16 42006 41990 131.206 111.188 0.0073127 0.00761657 2024-03-24T11:21:27.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:27.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:27.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:27.211 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:27.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:27.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:27.342 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:27.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:27.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:27.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:27 smithi179 ceph-mon[29608]: pgmap v77: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:27.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:27 smithi179 ceph-mon[37907]: pgmap v77: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:27.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:27.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:27.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:27.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:27.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:27.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:27 smithi181 ceph-mon[33998]: pgmap v77: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:27.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:27.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:21:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:27.761 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:27.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:27.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:27.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:27.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:28.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:28.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 21 16 43660 43644 129.88 103.375 0.0147973 0.00769412 2024-03-24T11:21:28.074 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:28.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:28.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:21:28.195 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:21:28.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-24T11:21:28.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:28.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:28.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:28.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:28.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:28.355 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:28.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:28.459 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:28 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[72689]: ts=2024-03-24T11:21:28.340Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:21:28.459 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:28 smithi179 podman[80541]: 2024-03-24 11:21:28.372468765 +0000 UTC m=+0.650687474 container died 4a895c887604272a0c1687c56a0936369db831c1f55bcc956c2cd2fd538d54a9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:28.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:28.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:28.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:28.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:28.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:28.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:28.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:28.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:28.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:28.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:28.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:28.950 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:28.971 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:29.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:29.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 22 16 45182 45166 128.299 95.125 0.0117731 0.00778849 2024-03-24T11:21:29.068 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:28 smithi179 podman[80541]: 2024-03-24 11:21:28.89695627 +0000 UTC m=+1.175174984 container cleanup 4a895c887604272a0c1687c56a0936369db831c1f55bcc956c2cd2fd538d54a9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:29.069 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:28 smithi179 bash[80541]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:21:29.118 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:29.166 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:29.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:29.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:29.307 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:29.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:29.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:29.447 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:29.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:29.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:29 smithi179 ceph-mon[29608]: pgmap v78: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:29.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:29 smithi179 ceph-mon[37907]: pgmap v78: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:29.459 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:29 smithi179 podman[80679]: 2024-03-24 11:21:29.208755408 +0000 UTC m=+0.831537445 container remove 4a895c887604272a0c1687c56a0936369db831c1f55bcc956c2cd2fd538d54a9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:29.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:29.581 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:21:29.587 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:29.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:29.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:29 smithi181 ceph-mon[33998]: pgmap v78: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-24T11:21:29.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:29.747 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:29.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:29.810 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:29 smithi179 podman[80679]: 2024-03-24 11:21:29.502133274 +0000 UTC m=+1.124915314 volume remove fe7c420dc93d9c9e42f69dde702437352f652d0674e312dd0053f449812e8f3c 2024-03-24T11:21:29.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:29.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:29.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:29.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:30.012 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:30.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:30.065 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 23 16 46753 46737 126.988 98.1875 0.00657387 0.00786958 2024-03-24T11:21:30.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:30.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:30.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:30.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:30.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:30.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:30.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:30.428 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:30.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:30.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:30.582 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:30.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:30.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:30.724 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:30.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:30.818 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:30 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:21:30.818 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:30 smithi179 systemd[1]: Stopped Ceph alertmanager.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:21:30.818 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:30 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.189s CPU time. 2024-03-24T11:21:30.818 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:30 smithi179 systemd[1]: Starting Ceph alertmanager.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:21:30.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:30.866 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:30.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:30.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:31.012 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:31.064 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 24 16 48332 48316 125.808 98.6875 0.0101586 0.00794386 2024-03-24T11:21:31.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:31.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:31.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:31.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:31.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:31.412 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:31 smithi179 ceph-mon[29608]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 864 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-24T11:21:31.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:31 smithi179 ceph-mon[37907]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 864 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-24T11:21:31.439 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:31.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:31.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:31.633 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:31.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:31 smithi181 ceph-mon[33998]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 864 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-24T11:21:31.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:31.701 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:31.706 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:31.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:31.848 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:31.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:31.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:31.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:32.065 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 25 16 49882 49866 124.651 96.875 0.00522187 0.00801751 2024-03-24T11:21:32.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:32.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:32.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:32.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:32.239 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:32.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:32.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:32.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:32.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:32.401 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:32 smithi179 podman[81185]: 2024-03-24 11:21:32.219419687 +0000 UTC m=+0.272784854 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-24T11:21:32.401 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:32.401 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:32.473 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:32.512 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:32.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:32.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:32.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:32.652 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:32.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:32.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:32.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:32.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:32.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:32.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:32.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:33.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:33.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:33.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:33.065 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 26 16 51365 51349 123.421 92.6875 0.0114254 0.00809738 2024-03-24T11:21:33.162 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:33.201 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:33.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:33.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:33.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:33.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:33.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:33 smithi181 ceph-mon[33998]: pgmap v80: 148 pgs: 148 active+clean; 2.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-24T11:21:33.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:21:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:21:33.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:33.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:33.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:33.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:33.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:33.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:33.628 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:33 smithi179 ceph-mon[29608]: pgmap v80: 148 pgs: 148 active+clean; 2.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-24T11:21:33.628 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:33 smithi179 ceph-mon[37907]: pgmap v80: 148 pgs: 148 active+clean; 2.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-24T11:21:33.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:33.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:33.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:33.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:33.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:33.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:33.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:34.028 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:34.065 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 27 15 52932 52917 122.479 98 0.0105107 0.00815952 2024-03-24T11:21:34.130 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:34.175 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:34.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:34.318 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:34.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:34.453 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:34.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:34.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:34.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:34.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:34.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:34.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:34.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:34.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:34.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:34.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:34.989 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:34.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:35.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 28 16 54395 54379 121.366 91.375 0.00639731 0.00823363 2024-03-24T11:21:35.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:35.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:35.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:35.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:35.185 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:21:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:35.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:35.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:35.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:35.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:35.383 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:35.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:35.469 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:35 smithi179 ceph-mon[29608]: pgmap v81: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-24T11:21:35.470 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:35 smithi179 ceph-mon[37907]: pgmap v81: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-24T11:21:35.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:35.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:35.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:35.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:35.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:35.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:35.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:35 smithi181 ceph-mon[33998]: pgmap v81: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-24T11:21:35.738 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:21:35.738 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:21:35.751 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:35.793 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:35.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:35.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:35.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:35.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:36.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:36.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 29 16 55809 55793 120.228 88.375 0.0122613 0.00831222 2024-03-24T11:21:36.074 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:36.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:36.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:36.217 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:36.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:36.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:36.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:36.352 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:36.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:36.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:36.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:36.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:36.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:36.608 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:36.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:36.690 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:36 smithi179 podman[81185]: 2024-03-24 11:21:36.372235143 +0000 UTC m=+4.425600274 volume create 1c5bb53b4443c6be6d0b3aed2597725e1df146bc61b12ddef09ec941f6673f78 2024-03-24T11:21:36.690 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:36 smithi179 podman[81185]: 2024-03-24 11:21:36.69016479 +0000 UTC m=+4.743529928 container create 12bd225c6028c54e350563a65647455f2c371ae8a0c5fec3f14eb54f12daf054 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:36.710 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:36.752 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:36.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:36.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:36.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:36.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:36.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:37.033 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:37.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 30 16 57241 57225 119.204 89.5 0.0109644 0.00838342 2024-03-24T11:21:37.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:37.177 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:37.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:37.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:37.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:21:37.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:21:37.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[29608]: pgmap v82: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T11:21:37.380 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[29608]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:21:37.381 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:21:37.381 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:21:37.381 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[37907]: pgmap v82: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T11:21:37.381 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-mon[37907]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:21:37.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.452 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:37.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:37.526 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:21:37.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:37.568 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:21:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.598 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:37.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:37.613 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:21:37.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:37 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:21:37.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:37 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:21:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:37 smithi181 ceph-mon[33998]: pgmap v82: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-24T11:21:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:37 smithi181 ceph-mon[33998]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T11:21:37.668 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:21:37.699 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:37.707 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:21:37.740 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:37.746 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:21:37.789 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:21:37.832 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:21:37.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:37.852 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:21:37.852 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:21:37.852 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:21:37.852 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:21:37.852 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:21:37.853 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:21:37.853 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:21:37.853 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:21:37.870 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:37.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 podman[81185]: 2024-03-24 11:21:37.77658037 +0000 UTC m=+5.829945522 container init 12bd225c6028c54e350563a65647455f2c371ae8a0c5fec3f14eb54f12daf054 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:37.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 podman[81185]: 2024-03-24 11:21:37.784027421 +0000 UTC m=+5.837392552 container start 12bd225c6028c54e350563a65647455f2c371ae8a0c5fec3f14eb54f12daf054 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:37.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.816Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:21:37.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.817Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:21:37.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.818Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.179 port=9094 2024-03-24T11:21:37.960 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.819Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:21:37.960 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.883Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:21:37.960 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.884Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:21:37.960 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.890Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:21:37.960 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:37 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:37.890Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:21:37.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:37.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:37.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:38.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 31 16 58424 58408 117.743 73.9375 0.0116031 0.00848825 2024-03-24T11:21:38.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:38.134 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:38.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:38.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:38.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:38.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:38.298 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:21:38.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:38.407 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:38.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:38.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:38.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:38.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:38.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:38.639 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:38.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:38 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/123749164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:38.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:38 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2816777306' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:21:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:38.683 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:38.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:38.692 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:38 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/123749164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:38.692 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:38 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2816777306' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:21:38.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:38 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/123749164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:38.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:38 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2816777306' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:21:38.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:38.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:38.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:38.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:38.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:38 smithi179 bash[81185]: 12bd225c6028c54e350563a65647455f2c371ae8a0c5fec3f14eb54f12daf054 2024-03-24T11:21:38.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:38 smithi179 systemd[1]: Started Ceph alertmanager.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:21:38.966 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:38.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:39.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 32 16 59785 59769 116.722 85.0625 0.00406125 0.00856112 2024-03-24T11:21:39.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:39.107 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:39.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:39.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:39.242 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:39.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:39.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:39.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:21:39.388 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:39.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:39.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:39.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:39.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:39.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:39 smithi181 ceph-mon[33998]: pgmap v84: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T11:21:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:39 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:21:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:39 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2816777306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:21:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:39 smithi181 ceph-mon[33998]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:21:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:39 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:39 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:39.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:39 smithi181 ceph-mon[33998]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:21:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:39.664 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:39.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:39.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[29608]: pgmap v84: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T11:21:39.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:21:39.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2816777306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:21:39.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[29608]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:21:39.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:39.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:39.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[29608]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:21:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[37907]: pgmap v84: 148 pgs: 148 active+clean; 3.1 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T11:21:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:21:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2816777306' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:21:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[37907]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T11:21:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-mon[37907]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T11:21:39.765 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:39.810 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:39.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:39.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:39.946 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:39.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:40.048 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:40.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 33 16 61116 61100 115.705 83.1875 0.0112505 0.00863824 2024-03-24T11:21:40.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:40.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:40.116 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:39 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:39.819Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000087547s 2024-03-24T11:21:40.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:40.219 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:40.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:40.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:40.358 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:40.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:40.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:40 smithi179 ceph-mon[29608]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:21:40.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:40 smithi179 ceph-mon[29608]: Reconfiguring daemon prometheus.a on smithi181 2024-03-24T11:21:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:40 smithi179 ceph-mon[37907]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:21:40.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:40 smithi179 ceph-mon[37907]: Reconfiguring daemon prometheus.a on smithi181 2024-03-24T11:21:40.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:40.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:40.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:40.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:40.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:40.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:40.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:40 smithi181 ceph-mon[33998]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:21:40.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:40 smithi181 ceph-mon[33998]: Reconfiguring daemon prometheus.a on smithi181 2024-03-24T11:21:40.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:40.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:40.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:40.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:40.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:40.903 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:40.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:41.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:41.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:41.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 34 16 62413 62397 114.686 81.0625 0.00524789 0.00871436 2024-03-24T11:21:41.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:41.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:41.213 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:41.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:41.362 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:21:41.363 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:21:41.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:41.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:21:41.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:21:41.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: pgmap v87: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:21:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:21:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:21:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:21:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:21:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:21:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:41 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:21:41.442 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: pgmap v87: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:21:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: pgmap v87: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:21:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:41 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:21:41.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:41.571 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:41.673 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:41.720 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:41.822 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:41.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:41.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:41.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:41.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:41.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:42.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 35 16 63530 63514 113.403 69.8125 0.0190279 0.00881146 2024-03-24T11:21:42.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:42.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:42.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:42.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:21:42.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:42.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:42.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:42.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:42.391 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:42.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:42.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:21:42.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:21:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:21:42.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:21:42.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:42 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:21:42.492 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:42.538 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:42.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:42.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:21:42.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:42.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:21:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:21:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:21:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:21:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:21:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:21:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:21:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:42 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:21:42.668 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:42.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:42.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:42.831 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:42.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:42.932 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:42.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:42.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:43.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 36 16 64474 64458 111.892 59 0.0154183 0.00893121 2024-03-24T11:21:43.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:43.116 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:43.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:43.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:43.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:43.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:43.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:43.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:43.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: pgmap v88: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:21:43.397 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:21:43.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:21:43.398 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:21:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:21:43.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:43.510 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:43.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:43.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:43.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:43.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: pgmap v88: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:21:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: pgmap v88: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:21:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:43 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:21:43.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:43.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:43.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:43.879 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:43.916 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:43.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:44.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:44.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:44.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:44.066 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 37 16 65596 65580 110.763 70.125 0.0106556 0.00902368 2024-03-24T11:21:44.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:44.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:44.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:44.293 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:44.336 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:44.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:44.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:44.473 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:44.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:44.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:44.617 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:44.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:44.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:21:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:21:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:21:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:21:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:21:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:21:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:44 smithi181 ceph-mon[33998]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:21:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:21:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:21:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:21:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[29608]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:21:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:44 smithi179 ceph-mon[37907]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T11:21:44.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:44.761 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:44.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:44.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:44.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:44.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:44.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:45.022 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:45.067 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 38 16 66700 66684 109.664 69 0.0201119 0.00911391 2024-03-24T11:21:45.124 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:45.162 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:45.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:45.305 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:45.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:45.336 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:21:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:45.336 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 systemd[1]: Stopping Ceph prometheus.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:21:45.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:45.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:45.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:45.562 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: pgmap v89: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 5.0 scrub starts 2024-03-24T11:21:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-mon[33998]: 5.0 scrub ok 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.422Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.422Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.422Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.422Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T11:21:45.653 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:21:45.654 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T11:21:45.654 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[62910]: ts=2024-03-24T11:21:45.423Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T11:21:45.654 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 podman[70102]: 2024-03-24 11:21:45.433154519 +0000 UTC m=+0.138678366 container died 354037eb0f72829c8ffb7c3136cefc62d10ba3bfb1801092d6602f9d3c9360e1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:45.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:45.707 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: pgmap v89: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 5.0 scrub starts 2024-03-24T11:21:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[29608]: 5.0 scrub ok 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: pgmap v89: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 5.0 scrub starts 2024-03-24T11:21:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:45 smithi179 ceph-mon[37907]: 5.0 scrub ok 2024-03-24T11:21:45.809 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.833 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:45.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:45.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:45.979 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:45.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:46.067 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 39 16 67728 67712 108.499 64.25 0.00986232 0.00921176 2024-03-24T11:21:46.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:46.084 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 podman[70102]: 2024-03-24 11:21:45.848589166 +0000 UTC m=+0.554112995 container cleanup 354037eb0f72829c8ffb7c3136cefc62d10ba3bfb1801092d6602f9d3c9360e1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:46.084 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:45 smithi181 bash[70102]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:21:46.113 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:46.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:46.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:46.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:46.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:46.349 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 podman[70130]: 2024-03-24 11:21:46.083797132 +0000 UTC m=+0.652957042 container remove 354037eb0f72829c8ffb7c3136cefc62d10ba3bfb1801092d6602f9d3c9360e1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:46.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:46.392 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:46.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:46.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:46.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:46.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:46.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:46.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:21:46.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:21:46.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:21:46.641 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:21:46.641 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T11:21:46.641 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 systemd[1]: Stopped Ceph prometheus.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:21:46.641 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.279s CPU time. 2024-03-24T11:21:46.641 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 systemd[1]: Starting Ceph prometheus.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:21:46.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:46.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:46.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:21:46.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:21:46.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:21:46.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:21:46.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:21:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:21:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:21:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:46 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:21:46.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:46.783 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:46.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:46.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:46.901 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 podman[70286]: 2024-03-24 11:21:46.544650446 +0000 UTC m=+0.019043459 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T11:21:46.901 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 podman[70286]: 2024-03-24 11:21:46.660359245 +0000 UTC m=+0.134752252 container create e7cb44f2b5d6e78c0b9f3fb922d1f0ccdcc6e3302b34970b37a0e3c128308253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:46.927 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:46.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:47.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:47.067 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:21:47.067742+0000 min lat: 0.00140901 max lat: 0.176217 avg lat: 0.00929648 2024-03-24T11:21:47.067 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:21:47.067 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 40 15 68831 68816 107.512 69 0.00714323 0.00929648 2024-03-24T11:21:47.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:47.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:47.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:47.211 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:47.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:47.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:47.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:47.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:47.352 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 podman[70286]: 2024-03-24 11:21:46.923590554 +0000 UTC m=+0.397983589 container init e7cb44f2b5d6e78c0b9f3fb922d1f0ccdcc6e3302b34970b37a0e3c128308253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:47.352 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 podman[70286]: 2024-03-24 11:21:46.927679946 +0000 UTC m=+0.402072969 container start e7cb44f2b5d6e78c0b9f3fb922d1f0ccdcc6e3302b34970b37a0e3c128308253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:47.352 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.965Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:21:47.352 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.965Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:21:47.352 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.965Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi181 (none))" 2024-03-24T11:21:47.352 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.965Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.965Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.967Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.968Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.969Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.969Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.973Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.973Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.838µs 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.973Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.974Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.974Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.974Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=70.995µs wal_replay_duration=744.163µs wbl_replay_duration=127ns total_replay_duration=1.097174ms 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.975Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:21:47.353 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.975Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:21:47.354 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:46.975Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:21:47.354 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:47.006Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=31.073854ms db_storage=874ns remote_storage=1.321µs web_handler=629ns query_engine=1.053µs scrape=625.53µs scrape_sd=88.77µs notify=17.65µs notify_sd=18.718µs rules=29.972798ms tracing=6.42µs 2024-03-24T11:21:47.354 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:47.007Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:21:47.354 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:21:47.007Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:21:47.354 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:47 smithi181 bash[70286]: e7cb44f2b5d6e78c0b9f3fb922d1f0ccdcc6e3302b34970b37a0e3c128308253 2024-03-24T11:21:47.354 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:21:47 smithi181 systemd[1]: Started Ceph prometheus.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:21:47.417 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STOPPING 2024-03-24T11:21:47.417 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:21:47.417 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STOPPED 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STARTING 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Serving on http://:::9283 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STARTED 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STOPPING 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STOPPED 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STARTING 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:21:47.418 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: pgmap v91: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 662 B/s rd, 49 MiB/s wr, 785 op/s 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 4.1d scrub starts 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 4.1d scrub ok 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi179.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:21:47.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: pgmap v91: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 662 B/s rd, 49 MiB/s wr, 785 op/s 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 4.1d scrub starts 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 4.1d scrub ok 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi179.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.422 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:47.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:47.509 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:47.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:47.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:47.625 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: pgmap v91: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 662 B/s rd, 49 MiB/s wr, 785 op/s 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 4.1d scrub starts 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 4.1d scrub ok 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:21:47.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi179.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:47.627 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:47.657 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:47.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:47.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Serving on http://:::9283 2024-03-24T11:21:47.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STARTED 2024-03-24T11:21:47.709 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STOPPING 2024-03-24T11:21:47.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:21:47.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STOPPED 2024-03-24T11:21:47.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STARTING 2024-03-24T11:21:47.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Serving on http://:::9283 2024-03-24T11:21:47.710 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:21:47] ENGINE Bus STARTED 2024-03-24T11:21:47.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:47.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:47.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:47.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:47.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:47.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:48.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:48.067 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 41 16 69907 69891 106.528 67.1875 0.0159563 0.00938249 2024-03-24T11:21:48.093 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:48.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:48.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:48.209 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:21:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:21:47.823Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003223658s 2024-03-24T11:21:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:48.240 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:48.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:48.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:48.378 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:48.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:48.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:48.519 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:48.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:48.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi179.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:21:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:48 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:21:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:48.665 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:48.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:21:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:21:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:21:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi179.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:21:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:21:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:21:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi179.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:21:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:21:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:21:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:21:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:21:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:21:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:21:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:48 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:21:48.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:48.812 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:48.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:48.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:48.952 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:48.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:49.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:49.068 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 42 16 71073 71057 105.724 72.875 0.0265009 0.00945371 2024-03-24T11:21:49.093 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:49.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:49.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:49.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:49.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:49.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:49.368 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:49.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:49.470 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:49.514 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:49.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:49.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: pgmap v93: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 758 op/s 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 4.0 scrub starts 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 4.0 scrub ok 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:21:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:49 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:21:49.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:49.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: pgmap v93: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 758 op/s 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 4.0 scrub starts 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 4.0 scrub ok 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:21:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: pgmap v93: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 758 op/s 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 4.0 scrub starts 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 4.0 scrub ok 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:21:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:49 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:21:49.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:49.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:49.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:49.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:49.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:49.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:50.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:50.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:50.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:50.068 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 43 15 72290 72275 105.036 76.125 0.0048236 0.00951585 2024-03-24T11:21:50.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:21:50.197 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:21:50.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-24T11:21:50.228 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:50.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:50.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:50.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:50.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:50.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:50.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:50.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:50.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:50.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:50.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:50.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:21:50.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:21:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 6.1b scrub starts 2024-03-24T11:21:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 6.1b scrub ok 2024-03-24T11:21:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:21:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:21:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:21:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:50 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:21:50.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:21:50.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:21:50.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 6.1b scrub starts 2024-03-24T11:21:50.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 6.1b scrub ok 2024-03-24T11:21:50.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:21:50.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 6.1b scrub starts 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 6.1b scrub ok 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:21:50.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:50 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:21:50.696 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:50.696 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:50.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:50.801 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:50.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:50.803 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:21:50.803 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:21:50.803 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:16:41.065262+0000", 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 309.72843432100001, 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.63059113300000003, 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14156", 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/4087802319", 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 1 2024-03-24T11:21:50.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065262+0000", 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065262+0000", 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.805 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065264+0000", 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3799999999999999e-06 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065269+0000", 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4500000000000003e-06 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.806 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065279+0000", 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5959999999999993e-06 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065289+0000", 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0155e-05 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065354+0000", 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5376999999999994e-05 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.807 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065370+0000", 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5548000000000001e-05 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.694719+0000", 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.629349241 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.694839+0000", 2024-03-24T11:21:50.808 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119639 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.694971+0000", 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131994 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695337+0000", 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00036660700000000002 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.809 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695489+0000", 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000151698 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695741+0000", 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025156199999999999 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695803+0000", 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1896000000000005e-05 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.810 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695854+0000", 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0989999999999998e-05 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:19520 7.b 7:dca6e7bf:::benchmark_data_smithi179_78267_object19519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:14.737129+0000", 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 36.056567907999998, 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033905717000000002, 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.811 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 19520 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.737129+0000", 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.737129+0000", 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.812 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.737130+0000", 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.13e-06 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.737149+0000", 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9293e-05 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.737165+0000", 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5228000000000001e-05 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.813 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.737172+0000", 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6179999999999997e-06 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.739500+0000", 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023280950000000001 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.739624+0000", 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000124227 2024-03-24T11:21:50.814 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.739783+0000", 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015863699999999999 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.741614+0000", 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001831222 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.741656+0000", 2024-03-24T11:21:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1869999999999997e-05 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.770927+0000", 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029270913999999999 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.770989+0000", 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2003999999999998e-05 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:14.771035+0000", 2024-03-24T11:21:50.816 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5479000000000003e-05 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:21045 7.b 7:d1a4101a:::benchmark_data_smithi179_78267_object21044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:15.397610+0000", 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 35.396087188999999, 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.039890673000000001, 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.817 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 21045 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397610+0000", 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397610+0000", 2024-03-24T11:21:50.818 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397612+0000", 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9149999999999999e-06 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397636+0000", 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4922999999999998e-05 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.819 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397655+0000", 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8076000000000001e-05 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397664+0000", 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8740000000000007e-06 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397728+0000", 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3720999999999995e-05 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.820 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397840+0000", 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111422 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.397975+0000", 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013554699999999999 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.399798+0000", 2024-03-24T11:21:50.821 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0018224420000000001 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.417794+0000", 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017996781999999999 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.437406+0000", 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019611659 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.822 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.437459+0000", 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3044000000000002e-05 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:15.437500+0000", 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1267999999999999e-05 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:37160 7.b 7:d16397dd:::benchmark_data_smithi179_78267_object37159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:21:50.823 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:24.228683+0000", 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 26.565013796999999, 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037588267000000002, 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 37160 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.824 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.228683+0000", 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.228683+0000", 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.825 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.228684+0000", 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3430000000000001e-06 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.228719+0000", 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4949e-05 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.228748+0000", 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8459e-05 2024-03-24T11:21:50.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.228760+0000", 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2458e-05 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229463+0000", 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00070258099999999997 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229589+0000", 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012631400000000001 2024-03-24T11:21:50.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229729+0000", 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014022000000000001 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.233342+0000", 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036122120000000001 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.237093+0000", 2024-03-24T11:21:50.828 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00375137 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.266082+0000", 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028988770000000001 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.266205+0000", 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012370099999999999 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.829 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.266271+0000", 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5889999999999994e-05 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:37164 7.b 7:d811e123:::benchmark_data_smithi179_78267_object37163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:24.229304+0000", 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 26.564392904000002, 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037710465999999998, 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.830 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 37164 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229304+0000", 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229304+0000", 2024-03-24T11:21:50.831 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229305+0000", 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4799999999999997e-07 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229327+0000", 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2035000000000002e-05 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229342+0000", 2024-03-24T11:21:50.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5075999999999999e-05 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.229350+0000", 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1759999999999993e-06 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.233375+0000", 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040253809999999998 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.233497+0000", 2024-03-24T11:21:50.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000121135 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.233638+0000", 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014164700000000001 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.237099+0000", 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0034610510000000001 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.834 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.238369+0000", 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0012699670000000001 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.266863+0000", 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.028493511999999999 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.266943+0000", 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0227000000000003e-05 2024-03-24T11:21:50.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:24.267014+0000", 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1410999999999998e-05 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:61794 7.b 7:dc304778:::benchmark_data_smithi179_78267_object61793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:40.559504+0000", 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 10.234192953000001, 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.03720205, 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.836 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 61794 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559504+0000", 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559504+0000", 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559506+0000", 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7940000000000001e-06 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559528+0000", 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2399999999999999e-05 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559544+0000", 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6232999999999999e-05 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559551+0000", 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8029999999999999e-06 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559612+0000", 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0967000000000001e-05 2024-03-24T11:21:50.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559730+0000", 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011812200000000001 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.559841+0000", 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110608 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.561843+0000", 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020017250000000002 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.562004+0000", 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000161535 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.596414+0000", 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034410198000000003 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.841 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.596652+0000", 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000238111 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:40.596706+0000", 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3554000000000003e-05 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:62747 7.b 7:daa15f3c:::benchmark_data_smithi179_78267_object62746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:21:50.842 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:41.341480+0000", 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.4522165240000007, 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.040718083000000002, 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 62747 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341480+0000", 2024-03-24T11:21:50.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341480+0000", 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341482+0000", 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.621e-06 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341508+0000", 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5933999999999999e-05 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341527+0000", 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8839999999999999e-05 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341537+0000", 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0292000000000001e-05 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341627+0000", 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9723999999999997e-05 2024-03-24T11:21:50.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341769+0000", 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014216399999999999 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.341921+0000", 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000152616 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.344225+0000", 2024-03-24T11:21:50.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0023035920000000001 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.352269+0000", 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0080442029999999998 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.382116+0000", 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029847167000000001 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.382160+0000", 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3863999999999999e-05 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.382198+0000", 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8065999999999998e-05 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:62749 7.b 7:d7b96b03:::benchmark_data_smithi179_78267_object62748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:41.345646+0000", 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.4480510130000006, 2024-03-24T11:21:50.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045221413000000002, 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 62749 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.345646+0000", 2024-03-24T11:21:50.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.345646+0000", 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.345648+0000", 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8330000000000001e-06 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.345674+0000", 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5976999999999999e-05 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.345693+0000", 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8941e-05 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.345703+0000", 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0212e-05 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.345775+0000", 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2286000000000006e-05 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346064+0000", 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028927500000000002 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346221+0000", 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000157052 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.349859+0000", 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036377699999999998 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.352278+0000", 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002419235 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.389141+0000", 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.036862248 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.389417+0000", 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027627800000000003 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.390867+0000", 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014503059999999999 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:62752 7.b 7:da963470:::benchmark_data_smithi179_78267_object62751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:21:50.855 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:41.346090+0000", 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 9.4476069729999992, 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044944118999999998, 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 62752 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346090+0000", 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346090+0000", 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346091+0000", 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5300000000000002e-07 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346113+0000", 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2674e-05 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346133+0000", 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9262e-05 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.346148+0000", 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5187999999999999e-05 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.349904+0000", 2024-03-24T11:21:50.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037561740000000001 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.350032+0000", 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128317 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.350179+0000", 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014699 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.352413+0000", 2024-03-24T11:21:50.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002233961 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.353905+0000", 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014916969999999999 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.390933+0000", 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037028024999999999 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.390996+0000", 2024-03-24T11:21:50.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2966999999999995e-05 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:41.391034+0000", 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7910999999999997e-05 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:64308 7.b 7:de14c4e2:::benchmark_data_smithi179_78267_object64307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:21:50.861 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.9321652599999997, 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.061113733000000003, 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 64308 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:21:50.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.15e-07 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861551+0000", 2024-03-24T11:21:50.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9168e-05 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861559+0000", 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7149999999999998e-06 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861565+0000", 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7209999999999999e-06 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868075+0000", 2024-03-24T11:21:50.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065101780000000001 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868193+0000", 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117935 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868332+0000", 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013922699999999999 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917732+0000", 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049399729000000003 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917778+0000", 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6468999999999998e-05 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922502+0000", 2024-03-24T11:21:50.866 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047234219999999997 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922598+0000", 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6825999999999998e-05 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922645+0000", 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6928000000000002e-05 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:64318 7.d e103/92 7:b3b8bb4b:::benchmark_data_smithi179_78267_object64317:head v 103'4007, mlcod=103'4005)", 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.868496+0000", 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.9252007689999999, 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048919531000000002, 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868496+0000", 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868496+0000", 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868497+0000", 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6400000000000001e-07 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868533+0000", 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5751999999999999e-05 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868544+0000", 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1012999999999999e-05 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868551+0000", 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1659999999999997e-06 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868621+0000", 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0265000000000006e-05 2024-03-24T11:21:50.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868639+0000", 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7689999999999998e-05 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917371+0000", 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048732343999999997 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917416+0000", 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4536999999999997e-05 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.871 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:64321 7.b 7:d4b02eee:::benchmark_data_smithi179_78267_object64320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.871087+0000", 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.9226100490000002, 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051668283000000002, 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 64321 2024-03-24T11:21:50.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871087+0000", 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871087+0000", 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871089+0000", 2024-03-24T11:21:50.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2350000000000002e-06 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871308+0000", 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021871299999999999 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871320+0000", 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2632e-05 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871329+0000", 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7399999999999993e-06 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915067+0000", 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043738251999999998 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915220+0000", 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015314699999999999 2024-03-24T11:21:50.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915916+0000", 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00069530899999999997 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922454+0000", 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065381079999999999 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922491+0000", 2024-03-24T11:21:50.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6823000000000003e-05 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922653+0000", 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016257700000000001 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922721+0000", 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8170999999999995e-05 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922755+0000", 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3575999999999998e-05 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:64319 7.9 e103/92 7:940c5710:::benchmark_data_smithi179_78267_object64318:head v 103'3958, mlcod=103'3956)", 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.871788+0000", 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.9219089799999995, 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045675184000000001, 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.878 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871788+0000", 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871788+0000", 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871789+0000", 2024-03-24T11:21:50.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5600000000000004e-07 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871815+0000", 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6478999999999998e-05 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871826+0000", 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.061e-05 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871832+0000", 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6540000000000002e-06 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915180+0000", 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.04334731 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915201+0000", 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0859000000000001e-05 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917431+0000", 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002229969 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917463+0000", 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2447000000000003e-05 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:64310 7.2 e103/92 7:4815ccc8:::benchmark_data_smithi179_78267_object64309:head v 103'4038, mlcod=103'4036)", 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.872567+0000", 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.9211300769999999, 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048127494999999999, 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872567+0000", 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872567+0000", 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872567+0000", 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6300000000000004e-07 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872591+0000", 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3328000000000001e-05 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914530+0000", 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041939428000000001 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914549+0000", 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8627999999999999e-05 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.885 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914784+0000", 2024-03-24T11:21:50.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00023493099999999999 2024-03-24T11:21:50.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914806+0000", 2024-03-24T11:21:50.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1979000000000001e-05 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.920472+0000", 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056665379999999996 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.920694+0000", 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0002219 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:64308 7.b e103/92 ondisk, result = 0)", 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.872744+0000", 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.9209529109999997, 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045041565999999998, 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872744+0000", 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872744+0000", 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872744+0000", 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1900000000000003e-07 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.872751+0000", 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3380000000000001e-06 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914908+0000", 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.042157027999999999 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914915+0000", 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.374e-06 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917768+0000", 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028527299999999999 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917773+0000", 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4820000000000002e-06 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917785+0000", 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2095e-05 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:64330 7.b 7:d19d208e:::benchmark_data_smithi179_78267_object64329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.888973+0000", 2024-03-24T11:21:50.891 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.9047241240000004, 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033973880999999997, 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 64330 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.888973+0000", 2024-03-24T11:21:50.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.888973+0000", 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.888975+0000", 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.554e-06 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.889002+0000", 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6295000000000002e-05 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914531+0000", 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025529676000000001 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.914550+0000", 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8386e-05 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917814+0000", 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032642869999999998 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917962+0000", 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014832100000000001 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.918111+0000", 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000149154 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922762+0000", 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0046505749999999997 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922794+0000", 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1628999999999998e-05 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922826+0000", 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2048000000000002e-05 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922908+0000", 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2751999999999996e-05 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922947+0000", 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8204000000000002e-05 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "rep_scrubmap(3.4 e65 from shard 2)", 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.963842+0000", 2024-03-24T11:21:50.897 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.8298546399999998, 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034416869000000003, 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.963842+0000", 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.963842+0000", 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.963843+0000", 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2900000000000004e-07 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.963846+0000", 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2229999999999999e-06 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.998160+0000", 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.034313801999999997 2024-03-24T11:21:50.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.998173+0000", 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2787e-05 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.998233+0000", 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0112000000000003e-05 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.998244+0000", 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1807999999999999e-05 2024-03-24T11:21:50.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.998259+0000", 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4608e-05 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:65602 7.b 7:d61d11e0:::benchmark_data_smithi179_78267_object65601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:44.073820+0000", 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 6.7198772819999997, 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.073232816000000006, 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.901 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 65602 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073820+0000", 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073820+0000", 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073821+0000", 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.562e-06 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073844+0000", 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2507999999999999e-05 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073856+0000", 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2463000000000001e-05 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073864+0000", 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8390000000000007e-06 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077258+0000", 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003393741 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077376+0000", 2024-03-24T11:21:50.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011797400000000001 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077466+0000", 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0828999999999994e-05 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143415+0000", 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065948735999999994 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:21:50.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143655+0000", 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000239349 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.144072+0000", 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00041780499999999999 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.146609+0000", 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025367800000000002 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.147052+0000", 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044323000000000002 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:65607 7.0 e104/92 7:0abd0400:::benchmark_data_smithi179_78267_object65606:head v 104'4199, mlcod=104'4197)", 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:44.077314+0000", 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 6.7163828810000004, 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.062365505000000002, 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:21:50.907 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077314+0000", 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077314+0000", 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077315+0000", 2024-03-24T11:21:50.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7999999999999995e-07 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077371+0000", 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6165000000000002e-05 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077378+0000", 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7449999999999998e-06 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077382+0000", 2024-03-24T11:21:50.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5759999999999999e-06 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077428+0000", 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5433999999999999e-05 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077441+0000", 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3534e-05 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.910 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.139579+0000", 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.062137723999999998 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.139679+0000", 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100647 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:65602 7.b e104/92 ondisk, result = 0)", 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 6.7145839780000003, 2024-03-24T11:21:50.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.064313464000000001, 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5799999999999999e-07 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079117+0000", 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.428e-06 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.137173+0000", 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058056435000000003 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.137184+0000", 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0368e-05 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143401+0000", 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062175820000000001 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143410+0000", 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6459999999999999e-06 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:21:50.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143426+0000", 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6447e-05 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:21:50.915 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:21:50.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:50.962 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:50.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:51.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:51.068 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 44 16 73322 73306 104.113 64.4375 0.00892329 0.00959952 2024-03-24T11:21:51.110 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:51.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:51.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:51.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:51.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:51.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:51.405 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:51.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:51.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:51.546 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:51.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:51.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:21:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:21:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: pgmap v95: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-24T11:21:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:21:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:21:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:21:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:21:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:21:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:21:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:21:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:51 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:21:51.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:51.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:21:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:21:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: pgmap v95: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-24T11:21:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:21:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: pgmap v95: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:21:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:21:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:21:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:21:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:21:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:21:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:21:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:51 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:21:51.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:51.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:51.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:51.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:51.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:51.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:52.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:52.068 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 45 16 74493 74477 103.426 73.1875 0.0122988 0.00966362 2024-03-24T11:21:52.103 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:52.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:52.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:52.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:52.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:52.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:52.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:52.377 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:52.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:52.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:52.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:52.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:52.615 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:52.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:21:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:52.657 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:52.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:21:52.705 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:21:52.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:52.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:52 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:52.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:52.806 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:52.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:21:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:21:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T11:21:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:21:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:21:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:21:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:21:52.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:21:52.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:21:52.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:52.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:52 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:21:52.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:52.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:52.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:53.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:53.069 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 46 16 75637 75621 102.73 71.5 0.00914853 0.00972885 2024-03-24T11:21:53.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:53.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:53.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:53.213 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:53.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:53.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:53.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:53.361 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:53.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:53.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:21:53.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:53.509 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:53.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:53.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:53.637 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:53.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:53.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:21:53.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:21:53.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[29608]: pgmap v97: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[37907]: pgmap v97: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:21:53.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:53 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:21:53.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:53.788 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:53.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:53.889 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:53.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:21:53.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:21:53.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-mon[33998]: pgmap v97: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 57 MiB/s wr, 906 op/s 2024-03-24T11:21:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:21:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:21:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:21:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:53 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:21:53.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:53.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:54.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:54.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:54.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:54.069 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 47 16 76707 76691 101.967 66.875 0.00690462 0.00980126 2024-03-24T11:21:54.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:54.201 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:54.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:54.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:54.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:54.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:54.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:54.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:54.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:54.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:54.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:54.615 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:54.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:21:54.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:21:54.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:54 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:21:54.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:54.767 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:54.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:54.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:54.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:54.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:21:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:21:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:21:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:21:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:21:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T11:21:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:21:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:54 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:21:54.932 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:55.000 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:55.057 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:55.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:55.072 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 48 16 77866 77850 101.35 72.4375 0.00633079 0.00986172 2024-03-24T11:21:55.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:55.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:55.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:55.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:21:55.335 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:55.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:21:55.427 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:21:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:55.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:55.475 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:55.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:55.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:55.634 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:55.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[29608]: pgmap v98: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[37907]: pgmap v98: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:21:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:55 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:21:55.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:55.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:55.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:55.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:55.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:55 smithi181 ceph-mon[33998]: pgmap v98: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T11:21:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:55 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:21:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:55 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:21:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:55 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:21:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:55 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:21:55.922 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:55.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:56.024 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:56.072 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 49 16 78994 78978 100.717 70.5 0.0319048 0.009923 2024-03-24T11:21:56.074 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:56.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:56.177 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:56.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:56.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:56.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:56.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:56.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:56.416 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:21:56.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:56.459 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:21:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:56.498 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:56.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:56.512 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:21:56.558 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1545777508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:21:56.576 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:21:56.577 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:21:56.577 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:56 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1545777508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:56.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:56.600 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:21:56.627 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:56.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:56.644 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:21:56.684 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:21:56.729 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:21:56.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:21:56.731 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:21:56.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:56.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:56.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:56.885 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:56.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:56.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:56 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:21:56.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:56 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:21:56.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:56 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:21:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:56 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:21:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:56 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1545777508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:21:56.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:57.012 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:57.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:57.075 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 50 16 79947 79931 99.8876 59.5625 0.00838863 0.0100068 2024-03-24T11:21:57.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:57.157 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:57.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:57.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:57.300 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:57.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:57.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:57.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:57.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:57.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:21:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:57.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:21:57.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: pgmap v100: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 732 B/s rd, 51 MiB/s wr, 819 op/s 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 4.19 scrub starts 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 4.19 scrub ok 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:21:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: pgmap v100: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 732 B/s rd, 51 MiB/s wr, 819 op/s 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 4.19 scrub starts 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 4.19 scrub ok 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:21:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:57 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:21:57.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:57.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:57.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:21:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:57.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:57.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:21:57.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: pgmap v100: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 732 B/s rd, 51 MiB/s wr, 819 op/s 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 4.19 scrub starts 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 4.19 scrub ok 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:21:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:57 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:21:57.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:58.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:58.075 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 51 16 80906 80890 99.1043 59.9375 0.0424522 0.0100836 2024-03-24T11:21:58.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:58.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:58.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:58.260 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:58.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:58.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:58.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:58.522 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:58.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.658 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:58.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:58.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:58 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:21:58.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:58 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:21:58.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:58 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:21:58.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:58 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:21:58.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:58.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:58.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:58 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:21:58.879 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:58 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:21:58.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:58.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:58.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:58.973 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:59.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:59.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:59.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:59.075 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 52 16 81904 81888 98.3972 62.375 0.0214515 0.0101572 2024-03-24T11:21:59.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:59.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:59.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:21:59.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:21:59.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:59.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:21:59.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:21:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:59.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:59.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:21:59.487 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:21:59.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:21:59.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:21:59.641 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:59.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: pgmap v101: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 716 op/s 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: 3.12 scrub starts 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: 3.12 scrub ok 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: 7.c scrub starts 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2826157177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:21:59.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: pgmap v101: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 716 op/s 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: 3.12 scrub starts 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: 3.12 scrub ok 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: 7.c scrub starts 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2826157177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:21:59.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:21:59 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:21:59.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:21:59.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:21:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:59.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:21:59.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:21:59.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: pgmap v101: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 716 op/s 2024-03-24T11:21:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: 3.12 scrub starts 2024-03-24T11:21:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: 3.12 scrub ok 2024-03-24T11:21:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: 7.c scrub starts 2024-03-24T11:21:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2826157177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:21:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:21:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:21:59 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:21:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:21:59.922 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:21:59.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:00.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:00.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:00.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:00.076 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 53 16 82921 82905 97.7397 63.5625 0.01684 0.0102259 2024-03-24T11:22:00.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:00.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:00.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:00.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:00.309 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:00.358 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:00.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:00.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:00.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:00.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:00.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:00.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:00.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: 7.7 scrub starts 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[29608]: 7.c scrub ok 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:22:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:22:00.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:22:00.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:22:00.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:22:00.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: 7.7 scrub starts 2024-03-24T11:22:00.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:22:00.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:00 smithi179 ceph-mon[37907]: 7.c scrub ok 2024-03-24T11:22:00.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:00.764 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:00.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:00.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:00.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:00.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:22:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:22:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:22:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:22:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:22:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: 7.7 scrub starts 2024-03-24T11:22:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T11:22:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:00 smithi181 ceph-mon[33998]: 7.c scrub ok 2024-03-24T11:22:00.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:01.041 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:01.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:01.076 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 54 16 83992 83976 97.1691 66.9375 0.0240132 0.0102868 2024-03-24T11:22:01.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:22:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:22:01.166 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:22:01.199 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:01.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:01.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:01.332 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:01.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:01.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:01.462 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:01.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:01.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:01.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:01.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:01.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:01.756 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:01.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:01.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: pgmap v103: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:22:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:22:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:22:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:22:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:22:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: 7.7 scrub ok 2024-03-24T11:22:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:22:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:01 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:22:01.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:01.905 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:01.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: pgmap v103: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:22:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: 7.7 scrub ok 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: pgmap v103: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:22:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: 7.7 scrub ok 2024-03-24T11:22:01.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:22:01.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:01 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:22:02.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:02.050 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:02.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:02.076 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 55 16 85063 85047 96.6195 66.9375 0.00632568 0.0103447 2024-03-24T11:22:02.152 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:02.194 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:02.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:02.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:02.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:02.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:02.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:02.478 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:02.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:02.580 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:02.621 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:02.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 5.1d deep-scrub starts 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 5.1d deep-scrub ok 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 3.8 deep-scrub starts 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 3.8 deep-scrub ok 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 7.8 scrub starts 2024-03-24T11:22:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:22:02.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:22:02.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:02.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:02.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:22:02.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:02 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:22:02.672 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:02.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:02.758 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:02.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:02.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 5.1d deep-scrub starts 2024-03-24T11:22:02.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 5.1d deep-scrub ok 2024-03-24T11:22:02.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:22:02.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:22:02.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 3.8 deep-scrub starts 2024-03-24T11:22:02.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 3.8 deep-scrub ok 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 7.8 scrub starts 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 5.1d deep-scrub starts 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 5.1d deep-scrub ok 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:22:02.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 3.8 deep-scrub starts 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 3.8 deep-scrub ok 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 7.8 scrub starts 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:22:02.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:02 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:22:02.852 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:22:02.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:02.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:02.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:02.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:03.022 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:03.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:03.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 56 16 86034 86018 95.9767 60.6875 0.0100952 0.0104131 2024-03-24T11:22:03.124 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:03.173 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:03.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:03.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:03.307 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:03.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:03.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:22:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:22:03.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:03.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:03.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:03.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:03.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:03.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:03.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:03.717 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:03.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:03.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:03 smithi179 ceph-mon[29608]: pgmap v104: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 659 op/s 2024-03-24T11:22:03.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:03 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:22:03.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:03 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:22:03.752 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:03 smithi179 ceph-mon[37907]: pgmap v104: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 659 op/s 2024-03-24T11:22:03.752 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:03 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:22:03.752 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:03 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:22:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:22:03.802 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:22:03.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-24T11:22:03.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:03.888 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:03.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:03 smithi181 ceph-mon[33998]: pgmap v104: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 659 op/s 2024-03-24T11:22:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:03 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:22:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:03 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:22:03.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:04.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:04.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 57 16 87050 87034 95.4069 63.5 0.00951753 0.0104756 2024-03-24T11:22:04.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.171 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:04.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:04.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:04.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:04.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:04.431 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:04.533 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.572 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:04.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:04.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.703 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:04.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:04.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:04.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:04 smithi181 ceph-mon[33998]: 7.8 scrub ok 2024-03-24T11:22:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:04 smithi181 ceph-mon[33998]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:22:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:04 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:22:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:04 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:22:04.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[29608]: 7.8 scrub ok 2024-03-24T11:22:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[29608]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:22:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:22:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:22:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[37907]: 7.8 scrub ok 2024-03-24T11:22:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[37907]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T11:22:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:22:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:04 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:22:04.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:04.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:04.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:05.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:05.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 58 16 88097 88081 94.8902 65.4375 0.00721815 0.0105335 2024-03-24T11:22:05.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:05.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:05.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:05.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:05.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:05.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:05.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:05.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:05.434 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:22:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:05.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:05.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:05.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:05.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:05.656 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:22:05.656 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:22:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:05.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:05.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:05.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:22:05.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:22:05.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[29608]: pgmap v105: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:22:05.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:22:05.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:22:05.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:22:05.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:22:05.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:22:05.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:22:05.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[37907]: pgmap v105: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:22:05.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:22:05.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:22:05.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:22:05.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:05 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:22:05.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:05.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:05.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:05 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:22:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:05 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:22:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:05 smithi181 ceph-mon[33998]: pgmap v105: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:22:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:05 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:22:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:05 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:22:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:05 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:22:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:05 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:22:05.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:05.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:05.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:06.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:06.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:06.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:06.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 59 16 89143 89127 94.3899 65.375 0.00607927 0.0105895 2024-03-24T11:22:06.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:06.217 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:06.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:06.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:06.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:06.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:06.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:06.503 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:06.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:06.604 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:06.646 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:06.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:06.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:06.792 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:06.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:06.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:06 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:22:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:06 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:22:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:06 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:22:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:06 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:22:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:06 smithi181 ceph-mon[33998]: 7.d scrub starts 2024-03-24T11:22:06.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:06.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[29608]: 7.d scrub starts 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:22:06.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:22:06.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:22:06.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:06 smithi179 ceph-mon[37907]: 7.d scrub starts 2024-03-24T11:22:07.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:22:07.037 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:07.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:22:07.078080+0000 min lat: 0.00140901 max lat: 0.176217 avg lat: 0.0106451 2024-03-24T11:22:07.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:22:07.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 60 16 90179 90163 93.8959 64.75 0.0108148 0.0106451 2024-03-24T11:22:07.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:07.095 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:07.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:07.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:07.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:07.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:07.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:07.365 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:07.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:07.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:07.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:07.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:07.599 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:07.668 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:07.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:07.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:07.837 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:07.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: 6.1c scrub starts 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: 6.1c scrub ok 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: 7.d scrub ok 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:22:07.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: 6.1c scrub starts 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: 6.1c scrub ok 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: 7.d scrub ok 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:22:07.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:07 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:22:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:22:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: 6.1c scrub starts 2024-03-24T11:22:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: 6.1c scrub ok 2024-03-24T11:22:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:22:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:22:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: 7.d scrub ok 2024-03-24T11:22:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:22:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:07 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:22:07.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:07.969 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:07.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:08.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:08.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 61 15 91162 91147 93.3648 61.5 0.0179099 0.0107054 2024-03-24T11:22:08.099 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:08.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:08.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:08.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:08.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:08.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:08.356 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:08.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:08.457 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:08.502 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:08.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:08.527 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:08.531 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:08.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:08.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:08.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:08.644 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:22:08.684 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:22:08.729 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:22:08.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:08.773 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:22:08.789 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:08.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:08.814 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:22:08.855 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:22:08.891 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:08.897 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:22:08.935 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:08.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:08.949 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:22:08.961 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:22:08.962 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:22:08.962 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:22:08.962 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:22:08.962 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:22:08.962 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:22:08.962 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:22:08.962 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:22:09.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:09.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:09.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:09.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 62 16 92067 92051 92.7703 56.5 0.0311575 0.0107742 2024-03-24T11:22:09.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:09.205 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:09.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:09.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:09.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:09.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:09.413 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:22:09.413 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:22:09.413 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: pgmap v108: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 599 B/s rd, 38 MiB/s wr, 601 op/s 2024-03-24T11:22:09.413 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/195282792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:22:09.414 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: pgmap v108: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 599 B/s rd, 38 MiB/s wr, 601 op/s 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:22:09.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/195282792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:09.437 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:09.478 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:09.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:09.579 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:09.616 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:09.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:09.643 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:09.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:22:09.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:22:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: pgmap v108: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 599 B/s rd, 38 MiB/s wr, 601 op/s 2024-03-24T11:22:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:22:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:22:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:22:09.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:22:09.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/195282792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:09.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:09.748 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:09.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:09.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:09.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:09.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:09.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:10.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:10.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 63 16 93051 93035 92.2739 61.5 0.00615314 0.0108321 2024-03-24T11:22:10.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:10.142 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:10.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:10.270 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:10.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:10.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.416 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:10.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:10.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[29608]: 3.1d deep-scrub starts 2024-03-24T11:22:10.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[29608]: 3.1d deep-scrub ok 2024-03-24T11:22:10.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[29608]: 6.e scrub starts 2024-03-24T11:22:10.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[29608]: 6.e scrub ok 2024-03-24T11:22:10.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[37907]: 3.1d deep-scrub starts 2024-03-24T11:22:10.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[37907]: 3.1d deep-scrub ok 2024-03-24T11:22:10.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[37907]: 6.e scrub starts 2024-03-24T11:22:10.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:10 smithi179 ceph-mon[37907]: 6.e scrub ok 2024-03-24T11:22:10.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:10.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:10.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:10 smithi181 ceph-mon[33998]: 3.1d deep-scrub starts 2024-03-24T11:22:10.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:10 smithi181 ceph-mon[33998]: 3.1d deep-scrub ok 2024-03-24T11:22:10.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:10 smithi181 ceph-mon[33998]: 6.e scrub starts 2024-03-24T11:22:10.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:10 smithi181 ceph-mon[33998]: 6.e scrub ok 2024-03-24T11:22:10.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:10.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:10.793 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.835 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:10.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:10.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:10.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:10.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:11.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:11.077 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 64 16 94077 94061 91.8341 64.125 0.00914996 0.0108845 2024-03-24T11:22:11.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:11.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:11.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:11.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:11.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:11.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:11.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:11.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:11.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: pgmap v109: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:22:11.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: pgmap v109: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:22:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:11 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:22:11.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:11.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:11.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:11.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:11.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:11.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:11.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:22:11.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:22:11.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: pgmap v109: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T11:22:11.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:22:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:22:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:22:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:22:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:22:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:11 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:22:11.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:11.753 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:11.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:11.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:11.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:11.882 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:11.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:11.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.021 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:12.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:12.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 65 16 95075 95059 91.3808 62.375 0.00406255 0.010938 2024-03-24T11:22:12.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:12.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:12.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:12.282 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:12.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:12.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:22:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:22:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:22:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:22:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:22:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:22:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:22:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:22:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:22:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:22:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:22:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:12 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:22:12.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.556 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:12.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:12.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:12 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:22:12.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:12 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:22:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:12 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:22:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:12 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:22:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:12 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:22:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:12 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:22:12.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:12.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:12.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:12.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:12.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:12.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:12.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:13.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:13.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:13.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:13.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 66 16 95921 95905 90.7974 52.875 0.00544476 0.0110084 2024-03-24T11:22:13.172 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:13.210 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:13.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:13.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:13.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:13.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:13.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[37907]: pgmap v110: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T11:22:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[37907]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:22:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[37907]: 7.9 scrub starts 2024-03-24T11:22:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:22:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:22:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[29608]: pgmap v110: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T11:22:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[29608]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:22:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[29608]: 7.9 scrub starts 2024-03-24T11:22:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:22:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:13 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:22:13.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:13.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:13.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:13.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:13.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:13 smithi181 ceph-mon[33998]: pgmap v110: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-24T11:22:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:13 smithi181 ceph-mon[33998]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T11:22:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:13 smithi181 ceph-mon[33998]: 7.9 scrub starts 2024-03-24T11:22:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:13 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:22:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:13 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:22:13.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:22:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:22:13.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:13.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:13.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:13.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:13.875 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:13.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:13.976 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:14.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:14.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:14.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 67 15 96778 96763 90.2426 53.625 0.0207625 0.0110761 2024-03-24T11:22:14.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:14.147 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:14.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:14.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:22:14.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:14.279 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:14.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:14.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:14.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:14.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:14 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:22:14.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:14 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:22:14.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:14 smithi181 ceph-mon[33998]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:22:14.437 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:14.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:22:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:22:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[29608]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:22:14.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:22:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:22:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:14 smithi179 ceph-mon[37907]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T11:22:14.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:14.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:14.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:14.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:14.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:14.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:14.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:14.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:14.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:14.956 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.001 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:15.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:15.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 68 16 97642 97626 89.7087 53.9375 0.0220384 0.0111424 2024-03-24T11:22:15.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:15.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:15.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:15.244 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:15 smithi179 ceph-mon[29608]: pgmap v112: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1 KiB/s rd, 62 MiB/s wr, 990 op/s 2024-03-24T11:22:15.245 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:15 smithi179 ceph-mon[29608]: 7.f scrub starts 2024-03-24T11:22:15.245 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:15 smithi179 ceph-mon[29608]: 7.9 scrub ok 2024-03-24T11:22:15.245 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:22:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:22:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:15.245 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:15 smithi179 ceph-mon[37907]: pgmap v112: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1 KiB/s rd, 62 MiB/s wr, 990 op/s 2024-03-24T11:22:15.245 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:15 smithi179 ceph-mon[37907]: 7.f scrub starts 2024-03-24T11:22:15.245 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:15 smithi179 ceph-mon[37907]: 7.9 scrub ok 2024-03-24T11:22:15.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:15.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:15.390 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:15.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:15.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:15.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:15.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:15.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:15.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:15 smithi181 ceph-mon[33998]: pgmap v112: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1 KiB/s rd, 62 MiB/s wr, 990 op/s 2024-03-24T11:22:15.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:15 smithi181 ceph-mon[33998]: 7.f scrub starts 2024-03-24T11:22:15.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:15 smithi181 ceph-mon[33998]: 7.9 scrub ok 2024-03-24T11:22:15.666 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:15.767 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:15.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:15.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:15.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:15.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:16.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:16.071 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:16.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:16.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 69 16 98588 98572 89.2655 59.125 0.0136513 0.0111973 2024-03-24T11:22:16.174 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:16.245 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:16.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:16.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:16.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:16.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:16.385 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:16 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:22:16.385 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:16 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:22:16.385 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:16 smithi179 ceph-mon[29608]: 7.b scrub starts 2024-03-24T11:22:16.385 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:16 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:22:16.385 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:16 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:22:16.385 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:16 smithi179 ceph-mon[37907]: 7.b scrub starts 2024-03-24T11:22:16.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:16.521 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:16.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:16.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:16.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:16 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:22:16.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:16 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:22:16.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:16 smithi181 ceph-mon[33998]: 7.b scrub starts 2024-03-24T11:22:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:16.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:16.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:16.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:16.780 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:16.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:16.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:16.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:16.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:17.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:17.056 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:17.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:17.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 70 16 99433 99417 88.7447 52.8125 0.0110286 0.0112633 2024-03-24T11:22:17.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:17.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:17.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:17.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:17.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:17.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:17.409 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:22:17.409 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:22:17.409 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-24T11:22:17.409 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:22:17.409 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:22:17.410 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:22:17.410 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:22:17.410 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:22:17.410 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[29608]: 7.f scrub ok 2024-03-24T11:22:17.410 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:22:17.411 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:22:17.411 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-24T11:22:17.411 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:22:17.411 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:22:17.411 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:22:17.411 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:22:17.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:22:17.412 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:17 smithi179 ceph-mon[37907]: 7.f scrub ok 2024-03-24T11:22:17.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:17.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:17.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:17.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:17.635 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:17.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:17.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T11:22:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:17 smithi181 ceph-mon[33998]: 7.f scrub ok 2024-03-24T11:22:17.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:17.763 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:17.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:17.865 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:17.915 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:17.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:18.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:18.068 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:18.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:18.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 71 16 100233 100217 88.199 50 0.0167141 0.0113325 2024-03-24T11:22:18.114 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:22:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:22:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:22:18.114 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:22:18.114 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:22:18.153 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:22:18.154 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:22:18.154 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:22:18.154 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:22:18.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:18.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:18.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:18.240 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:22:18.258 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[29608]: 3.18 scrub starts 2024-03-24T11:22:18.258 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[29608]: 3.18 scrub ok 2024-03-24T11:22:18.259 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:18.259 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[29608]: 7.4 scrub starts 2024-03-24T11:22:18.259 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2807623567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:22:18.259 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[29608]: 7.b scrub ok 2024-03-24T11:22:18.259 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[37907]: 3.18 scrub starts 2024-03-24T11:22:18.259 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[37907]: 3.18 scrub ok 2024-03-24T11:22:18.259 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:18.260 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[37907]: 7.4 scrub starts 2024-03-24T11:22:18.260 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2807623567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:22:18.260 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:18 smithi179 ceph-mon[37907]: 7.b scrub ok 2024-03-24T11:22:18.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:18.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:18.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:18.430 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:18.468 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:18.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:18.558 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:18.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:18.627 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:18.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:18.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:18 smithi181 ceph-mon[33998]: 3.18 scrub starts 2024-03-24T11:22:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:18 smithi181 ceph-mon[33998]: 3.18 scrub ok 2024-03-24T11:22:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:18 smithi181 ceph-mon[33998]: 7.4 scrub starts 2024-03-24T11:22:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:18 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2807623567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:22:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:18 smithi181 ceph-mon[33998]: 7.b scrub ok 2024-03-24T11:22:18.728 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:18.770 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:18.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:18.871 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:18.906 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:18.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:19.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:19.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:19.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:19.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 72 16 101269 101253 87.8733 64.75 0.0202506 0.0113753 2024-03-24T11:22:19.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:19.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:19.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:19.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:19.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:22:19.317 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:19.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:19.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:19.449 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:19.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:19.526 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[29608]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 792 op/s 2024-03-24T11:22:19.526 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:19.526 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2807623567' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:22:19.526 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[29608]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:22:19.526 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[37907]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 792 op/s 2024-03-24T11:22:19.527 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:19.527 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2807623567' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:22:19.527 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:19 smithi179 ceph-mon[37907]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:22:19.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:19.603 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:19.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:19.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:19.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:19 smithi181 ceph-mon[33998]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 792 op/s 2024-03-24T11:22:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:19 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2807623567' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:22:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:19 smithi181 ceph-mon[33998]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T11:22:19.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:19.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:19.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:19.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:19.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:19.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:22:19.901 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:22:19.901 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-24T11:22:19.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:20.031 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:20.078 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 73 16 102244 102228 87.5043 60.9375 0.00989971 0.0114227 2024-03-24T11:22:20.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:20.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:20.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:20.293 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:20.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:20.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:20.527 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.578 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:20.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:20.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:20.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:20.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:20.841 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:20.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:20.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:20.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:21.042 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:21.056 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:21.072 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:21.080 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 74 16 103242 103226 87.1628 62.375 0.0266725 0.0114673 2024-03-24T11:22:21.142 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:21.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:21.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:21.272 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:21.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:21.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:21.407 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:21.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:21.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:21.538 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:21.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:21.565 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:22:21.565 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:22:21.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[37907]: pgmap v118: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 45 MiB/s wr, 724 op/s 2024-03-24T11:22:21.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[37907]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:22:21.567 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:22:21.567 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:22:21.568 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[29608]: pgmap v118: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 45 MiB/s wr, 724 op/s 2024-03-24T11:22:21.568 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:21 smithi179 ceph-mon[29608]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:22:21.640 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:21.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:21 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:22:21.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:21 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:22:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:21 smithi181 ceph-mon[33998]: pgmap v118: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 45 MiB/s wr, 724 op/s 2024-03-24T11:22:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:21 smithi181 ceph-mon[33998]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T11:22:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:21.682 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:21.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:21.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:21.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:21.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:21.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:21.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:21.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:22.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:22.080 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 75 16 104302 104286 86.8839 66.25 0.0149349 0.0115048 2024-03-24T11:22:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:22.105 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:22.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:22.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:22.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:22.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:22.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:22.351 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:22:22.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:22.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:22.403 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:22:22.441 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:22:22.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:22.484 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:22:22.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:22.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:22.526 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:22:22.564 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:22:22.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:22.601 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:22:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:22.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:22.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:22.634 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:22:22.635 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:22:22.635 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:22:22.635 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:22:22.635 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:22:22.635 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:22:22.636 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:22:22.636 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:22:22.636 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:22:22.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:22.767 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:22.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:22.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:22.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:22.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:22.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:23.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:23.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:23.080 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 76 16 105279 105263 86.544 61.0625 0.0170844 0.0115494 2024-03-24T11:22:23.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:23.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:23.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:23.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:23.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:23.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:23.375 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:23 smithi179 ceph-mon[29608]: pgmap v120: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 850 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-24T11:22:23.376 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3941053472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:23.376 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:23 smithi179 ceph-mon[37907]: pgmap v120: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 850 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-24T11:22:23.376 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3941053472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:23.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:23.447 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:23.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:23.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:23.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:23.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:23.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:22:23.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:23 smithi181 ceph-mon[33998]: pgmap v120: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 850 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-24T11:22:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3941053472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:23.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:22:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:22:23.698 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:23.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:23.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:23.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:23.896 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:23.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:23.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:24.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:24.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 77 16 106267 106251 86.2215 61.75 0.00490977 0.0115929 2024-03-24T11:22:24.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.166 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:24.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:24.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:24.310 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:24.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:24.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:24.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.585 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:24.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:24.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:24.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:24.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:24.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:24.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:24.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:24.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:24.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:25.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:25.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 78 16 107337 107321 85.9735 66.875 0.0108048 0.0116271 2024-03-24T11:22:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:25.120 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:25.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:25.181 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:22:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:22:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:25.221 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:25.280 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:25.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:25.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:25.423 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:25.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:25.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:25 smithi179 ceph-mon[29608]: pgmap v121: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-24T11:22:25.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:25 smithi179 ceph-mon[37907]: pgmap v121: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-24T11:22:25.524 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:25.570 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:25.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:25.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:25 smithi181 ceph-mon[33998]: pgmap v121: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-24T11:22:25.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:25.702 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:25.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:25.804 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:25.843 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:25.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:25.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:25.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:25.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:26.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:26.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 79 15 108328 108313 85.67 62 0.00756966 0.0116679 2024-03-24T11:22:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:26.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:26.126 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:26.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:26.264 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:26.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:26.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:22:26.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:26.392 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:26.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:26.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:26.577 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:26.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:26.665 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:22:26.665 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:22:26.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:26.727 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:26.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:26.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:26.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:26.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:26.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:26.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:26.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:27.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:22:27.081984+0000 min lat: 0.00140901 max lat: 0.176217 avg lat: 0.0117311 2024-03-24T11:22:27.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:22:27.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 80 16 109108 109092 85.2077 48.6875 0.0172985 0.0117311 2024-03-24T11:22:27.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:27.116 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:27.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:27.217 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:27.257 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:27.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:27.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:27.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:27.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:27.424 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:27 smithi179 ceph-mon[29608]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:22:27.424 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:27 smithi179 ceph-mon[37907]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:22:27.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:27.527 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:27.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:27.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:27.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:27 smithi181 ceph-mon[33998]: pgmap v122: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:22:27.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:27.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:27.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:27.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:27.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:27.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:27.916 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:27.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:28.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:28.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:28.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 81 15 109986 109971 84.834 54.9375 0.0198334 0.0117831 2024-03-24T11:22:28.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:28.175 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:28.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:28.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:28.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.445 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:28.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:28.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:28.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:28.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:28.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:28.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.835 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:28.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:28.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:28.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:28.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:29.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:29.081 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 82 16 110985 110969 84.5601 62.375 0.0081678 0.0118216 2024-03-24T11:22:29.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:22:29.097 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:29.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:29.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:29.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:29.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:29.231 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:22:29.231 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:22:29.232 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:22:29.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:29.383 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:29.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:29.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:29 smithi179 ceph-mon[29608]: pgmap v123: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 36 MiB/s wr, 572 op/s 2024-03-24T11:22:29.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:29 smithi179 ceph-mon[37907]: pgmap v123: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 36 MiB/s wr, 572 op/s 2024-03-24T11:22:29.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:29.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:29.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:29.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:29.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:29 smithi181 ceph-mon[33998]: pgmap v123: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 36 MiB/s wr, 572 op/s 2024-03-24T11:22:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:29.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:29.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:29.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:29.803 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:29.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:29.905 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:29.943 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:29.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:30.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:30.073 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:30.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:30.084 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 83 16 111999 111983 84.3024 63.375 0.017505 0.0118567 2024-03-24T11:22:30.175 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:30.223 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:30.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:30.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:30.389 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:30.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:30.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:30.538 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:30.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:30.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:30.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:30.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:30.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:30.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:30.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:30.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:30.953 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:30.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:31.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:31.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 84 16 113060 113044 84.0881 66.3125 0.01016 0.011887 2024-03-24T11:22:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:31.100 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:31.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:31.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:31.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:31.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:31.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:31.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:31.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:31.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:31.513 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:31.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:31.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:31.639 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:31.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:31.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:31 smithi181 ceph-mon[33998]: pgmap v124: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1021 B/s rd, 61 MiB/s wr, 974 op/s 2024-03-24T11:22:31.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:31 smithi179 ceph-mon[29608]: pgmap v124: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1021 B/s rd, 61 MiB/s wr, 974 op/s 2024-03-24T11:22:31.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:31 smithi179 ceph-mon[37907]: pgmap v124: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1021 B/s rd, 61 MiB/s wr, 974 op/s 2024-03-24T11:22:31.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:31.779 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:31.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:31.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:31.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:31.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:32.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:32.047 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:32.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:32.084 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 85 16 114105 114089 83.8672 65.3125 0.0105336 0.0119184 2024-03-24T11:22:32.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:32.185 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:32.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:32.286 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:22:32.303 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:22:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:32.349 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:32.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:32.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:32.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:32.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:32.535 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:32.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:32.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:32.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:32.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:32.758 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:32.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:32.795 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:32.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:32.911 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:32.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:33.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:33.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.057 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:33.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:33.084 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 86 16 115035 115019 83.5679 58.125 0.0121496 0.0119615 2024-03-24T11:22:33.159 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.197 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:33.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:33.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.344 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:33.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:33.346 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:22:33.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:33.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:33.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:33.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:33.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:33 smithi181 ceph-mon[33998]: pgmap v125: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 867 B/s rd, 52 MiB/s wr, 827 op/s 2024-03-24T11:22:33.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:33.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:33.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/685689515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:22:33.653 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:22:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:22:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[29608]: pgmap v125: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 867 B/s rd, 52 MiB/s wr, 827 op/s 2024-03-24T11:22:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/685689515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:22:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[37907]: pgmap v125: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 867 B/s rd, 52 MiB/s wr, 827 op/s 2024-03-24T11:22:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:22:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/685689515' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:22:33.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:33.759 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:33.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:33.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:33.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:33.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:34.032 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:34.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:34.084 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 87 16 116040 116024 83.3293 62.8125 0.00825806 0.0119957 2024-03-24T11:22:34.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:34.173 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:34.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:34.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:34.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:34.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:34.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:34.448 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:34.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:34.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:34.579 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:34.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:34.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:34 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:34 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/685689515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:22:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:34 smithi181 ceph-mon[33998]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:22:34.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:34.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:34.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/685689515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:22:34.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[29608]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:22:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/685689515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:22:34.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:34 smithi179 ceph-mon[37907]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T11:22:34.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:34.745 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:34.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:34.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:34.876 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:34.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:34.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:35.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:35.084 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 88 16 117090 117074 83.1281 65.625 0.00598671 0.0120246 2024-03-24T11:22:35.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:35.150 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:35.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:35.283 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:35.329 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:22:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:22:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:35.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:35.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:35.514 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:35.557 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:35.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:35 smithi181 ceph-mon[33998]: pgmap v127: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:22:35.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:35.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:35 smithi179 ceph-mon[29608]: pgmap v127: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:22:35.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:35 smithi179 ceph-mon[37907]: pgmap v127: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:22:35.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.832 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:35.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:35.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:35.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:35.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:36.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:36.084 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 89 16 118134 118118 82.9271 65.25 0.0100255 0.012054 2024-03-24T11:22:36.096 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:36.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:36.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:36.234 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:36.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:36.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:36.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:36.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:36.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:36.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:36.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:36.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:36.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:36.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:36.791 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:36.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:36.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:36.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:36.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:37.025 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:37.069 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:37.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:37.084 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total time run: 90.0166 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total writes made: 119191 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Write size: 65536 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object size: 65536 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Bandwidth (MB/sec): 82.7563 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Bandwidth: 30.9811 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max bandwidth (MB/sec): 185.562 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min bandwidth (MB/sec): 48.6875 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average IOPS: 1324 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev IOPS: 495.698 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max IOPS: 2969 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min IOPS: 779 2024-03-24T11:22:37.085 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average Latency(s): 0.012079 2024-03-24T11:22:37.086 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Latency(s): 0.00879385 2024-03-24T11:22:37.086 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max latency(s): 0.176217 2024-03-24T11:22:37.086 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min latency(s): 0.00140901 2024-03-24T11:22:37.092 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T11:22:37.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:37.214 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:37.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:37.304 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:22:37.304 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:22:37.305 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-24T11:22:37.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-24T11:22:37.317 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:37.399 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:37.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:37.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:22:37.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:37.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:37.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:37.596 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[29608]: pgmap v128: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:22:37.597 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:22:37.597 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:22:37.597 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[29608]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:22:37.597 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[37907]: pgmap v128: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:22:37.597 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:22:37.597 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:22:37.597 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:37 smithi179 ceph-mon[37907]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:22:37.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:37.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:37 smithi181 ceph-mon[33998]: pgmap v128: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T11:22:37.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:37 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:22:37.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:37 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:22:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:37 smithi181 ceph-mon[33998]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T11:22:37.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:37.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:37.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:37.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:37.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:37.837 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:22:37.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:37.896 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:22:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:37.933 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:37.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:37.938 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:22:37.984 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:22:38.026 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:22:38.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:38.062 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:22:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:38.079 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:38.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:38.102 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:22:38.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:22:38.162 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:22:38.162 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:22:38.162 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:22:38.163 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:22:38.163 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:22:38.163 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:22:38.163 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:22:38.163 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:22:38.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:38.230 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:120186 7.f e117/92 7:fa7f4490:::benchmark_data_smithi179_78267_object992:head v 117'7552, mlcod=117'7550)", 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.213329+0000", 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.013871865000000001, 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013904372, 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:22:38.231 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.213329+0000", 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.213329+0000", 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.213329+0000", 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.41e-07 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.213333+0000", 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1200000000000004e-06 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.213340+0000", 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3999999999999997e-06 2024-03-24T11:22:38.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.213344+0000", 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8679999999999998e-06 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.216206+0000", 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028617249999999999 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.216226+0000", 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9783999999999999e-05 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:120196 7.6 e117/92 7:62e45f8a:::benchmark_data_smithi179_78267_object1002:head v 117'7638, mlcod=117'7636)", 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.219784+0000", 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0074170440000000002, 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0075149910000000004, 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.235 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.219784+0000", 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.219784+0000", 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.219784+0000", 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7000000000000003e-07 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.219792+0000", 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3019999999999997e-06 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.219803+0000", 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0715e-05 2024-03-24T11:22:38.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.219810+0000", 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.379e-06 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220019+0000", 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020937399999999999 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:22:38.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220039+0000", 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9656999999999998e-05 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:120198 7.3 7:c40ef1b7:::benchmark_data_smithi179_78267_object1004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.220241+0000", 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.006959888, 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071276200000000003, 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:22:38.239 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 120198 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220241+0000", 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220241+0000", 2024-03-24T11:22:38.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220242+0000", 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4810000000000001e-06 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220248+0000", 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9780000000000002e-06 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220261+0000", 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3165e-05 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220269+0000", 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8290000000000008e-06 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220350+0000", 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0428999999999999e-05 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220462+0000", 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112163 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220575+0000", 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011355 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.223756+0000", 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031807049999999998 2024-03-24T11:22:38.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227320+0000", 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003564322 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:120199 7.2 7:401a8198:::benchmark_data_smithi179_78267_object1005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.220288+0000", 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0069128519999999997, 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071712269999999996, 2024-03-24T11:22:38.244 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 120199 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220288+0000", 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220288+0000", 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220288+0000", 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0699999999999998e-07 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220291+0000", 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4849999999999999e-06 2024-03-24T11:22:38.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220297+0000", 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2110000000000003e-06 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.220302+0000", 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1909999999999999e-06 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:22:38.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.222577+0000", 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022746860000000002 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.222705+0000", 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012849000000000001 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.222817+0000", 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011188099999999999 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:22:38.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.226688+0000", 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00387134 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:120200 7.0 e117/92)", 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.226970+0000", 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.000230468, 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056262599999999999, 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.226970+0000", 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.226970+0000", 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.226971+0000", 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4000000000000001e-07 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.226978+0000", 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7540000000000006e-06 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.226991+0000", 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2904999999999999e-05 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.226998+0000", 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3999999999999997e-06 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:120188 7.f e117/92 7:fb16e8e3:::benchmark_data_smithi179_78267_object994:head v 117'7553, mlcod=117'7550)", 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.227021+0000", 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00017957699999999999, 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00055967899999999999, 2024-03-24T11:22:38.252 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227021+0000", 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227021+0000", 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227021+0000", 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1899999999999998e-07 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227028+0000", 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8889999999999996e-06 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227035+0000", 2024-03-24T11:22:38.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6579999999999996e-06 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227040+0000", 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7600000000000002e-06 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227132+0000", 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1954999999999996e-05 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:22:38.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227150+0000", 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8479e-05 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:120203 7.3 7:c732a571:::benchmark_data_smithi179_78267_object1009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.227507+0000", 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4294967295.9996939, 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013462700000000001, 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:22:38.256 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 120203 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227507+0000", 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227507+0000", 2024-03-24T11:22:38.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227508+0000", 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5039999999999999e-06 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227513+0000", 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3909999999999997e-06 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227528+0000", 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4626e-05 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227537+0000", 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1439999999999992e-06 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227585+0000", 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8171999999999999e-05 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.259 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:120199 7.2 e117/92)", 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:38.227633+0000", 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4294967295.9995675, 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4258999999999996e-05, 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227633+0000", 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227633+0000", 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227634+0000", 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6000000000000003e-07 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227639+0000", 2024-03-24T11:22:38.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9509999999999997e-06 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227649+0000", 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0443e-05 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:38.227656+0000", 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6100000000000002e-06 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:22:38.262 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.263 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:22:38.263 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:22:38.263 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 8 2024-03-24T11:22:38.263 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:22:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:38.264 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:38.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:38.365 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:38.401 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:38.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:38.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:38 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2890876668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:38 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2890876668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:38.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:38.531 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:38.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:38.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:38 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2890876668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:38.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:38.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:38.763 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:38.803 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:38.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:38.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:38.934 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:38.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:39.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:39.076 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:39.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:39.178 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:39.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:39.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:39.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:39.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:39.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:39.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:39.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:39 smithi179 ceph-mon[37907]: pgmap v130: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 748 op/s 2024-03-24T11:22:39.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:39 smithi179 ceph-mon[29608]: pgmap v130: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 748 op/s 2024-03-24T11:22:39.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:39.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:39.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:39.626 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:39.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:39.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:39 smithi181 ceph-mon[33998]: pgmap v130: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 748 op/s 2024-03-24T11:22:39.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:39.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:39.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:39.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:39.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:39.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:39.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:40.029 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:40.082 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:22:40.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:40.136 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:40.136 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:40.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:40.259 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:40.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:40.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:40.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:40.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:40.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:40.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:40.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:40.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.853 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:40.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:40.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:40.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:40.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:41.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:41.126 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:41.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:41.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:41.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:41.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:41.397 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:41.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:41.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:41.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:41.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:41.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:22:41.628 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:41 smithi179 ceph-mon[29608]: pgmap v131: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 72 MiB/s wr, 1.26k op/s 2024-03-24T11:22:41.629 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:41 smithi179 ceph-mon[37907]: pgmap v131: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 72 MiB/s wr, 1.26k op/s 2024-03-24T11:22:41.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:41.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:41 smithi181 ceph-mon[33998]: pgmap v131: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 72 MiB/s wr, 1.26k op/s 2024-03-24T11:22:41.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:41.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:41.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:41.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:41.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:41.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:41.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:41.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:41.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:42.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:42.056 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:42.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:42.157 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:42.222 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:42.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:42.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:42.369 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:42.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:42.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:42.500 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:42.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:42.596 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:42.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:42.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:42.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:42.689 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:42.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:42.783 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:42.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:42.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:42.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:42.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:43.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:43.082 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:43.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:43.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:43.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:43.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:43.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:43.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:43.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:43.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:43.490 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:43.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:43.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:43.619 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:43.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:43.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:43 smithi181 ceph-mon[33998]: pgmap v132: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 704 B/s rd, 29 MiB/s wr, 580 op/s 2024-03-24T11:22:43.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:22:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:22:43.681 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:43 smithi179 ceph-mon[29608]: pgmap v132: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 704 B/s rd, 29 MiB/s wr, 580 op/s 2024-03-24T11:22:43.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:43 smithi179 ceph-mon[37907]: pgmap v132: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 704 B/s rd, 29 MiB/s wr, 580 op/s 2024-03-24T11:22:43.720 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:43.762 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:43.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:43.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:43.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:43.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:43.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:44.039 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:44.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:44.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:44.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:44.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:44.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:44.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:44.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:44.366 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:22:44.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:44.455 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:44.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:44.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:44.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:44.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:44.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/523498532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:22:44.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/523498532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:22:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/523498532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:22:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:44.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:44.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:44.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:44.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:44.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:44.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.005 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:45.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:45.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:45.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:45.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:45.279 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:45.363 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:22:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:45.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.425 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:45.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:45.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:45.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:45.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:45 smithi181 ceph-mon[33998]: pgmap v133: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 1.02k op/s 2024-03-24T11:22:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:45 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:22:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:45 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:22:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:45 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:45 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/523498532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:22:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:45 smithi181 ceph-mon[33998]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:22:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:45 smithi181 ceph-mon[33998]: 7.e scrub starts 2024-03-24T11:22:45.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:45.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[29608]: pgmap v133: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 1.02k op/s 2024-03-24T11:22:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:22:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:22:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/523498532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:22:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[29608]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:22:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[29608]: 7.e scrub starts 2024-03-24T11:22:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[37907]: pgmap v133: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 1.02k op/s 2024-03-24T11:22:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:22:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:22:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:22:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/523498532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:22:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[37907]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T11:22:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:45 smithi179 ceph-mon[37907]: 7.e scrub starts 2024-03-24T11:22:45.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:45.822 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:45.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:45.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:45.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:46.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:46.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:46.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:46.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:46.234 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:46.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:46.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:46.361 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:46.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:46.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:46.493 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:46.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:46.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:46.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:46.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:46.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:46.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:46.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:46.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:46.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:46.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:46.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:47.036 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:47.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:47.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:47.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:47.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:47.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:47.440 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:47.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:47.581 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:47.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:47.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:47 smithi181 ceph-mon[33998]: pgmap v135: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 1.05k op/s 2024-03-24T11:22:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:47 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:22:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:47 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:22:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:47 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:22:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:47 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:22:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:47.683 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:47.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[29608]: pgmap v135: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 1.05k op/s 2024-03-24T11:22:47.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:22:47.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:22:47.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:22:47.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:22:47.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[37907]: pgmap v135: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 MiB/s wr, 1.05k op/s 2024-03-24T11:22:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:22:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:22:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:22:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:22:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.723 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:47.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:47.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:47.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:47.954 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:47.995 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:47.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:47.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:48.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:48.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:48.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:48.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:48.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:48.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:48.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:48.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:48.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:22:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:22:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:22:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:22:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[29608]: 7.e scrub ok 2024-03-24T11:22:48.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:22:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:22:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:22:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:22:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:48 smithi179 ceph-mon[37907]: 7.e scrub ok 2024-03-24T11:22:48.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:48.548 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:48.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:48.649 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:48 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:22:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:48 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:22:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:48 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:22:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:48 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:22:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:48 smithi181 ceph-mon[33998]: 7.e scrub ok 2024-03-24T11:22:48.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:48.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:48.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:48.826 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:48.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:48.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:48.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:48.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:48.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:49.019 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:22:49.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:49.058 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:22:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:49.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:49.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:49.101 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:22:49.140 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:22:49.175 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:22:49.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:49.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:49.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:49.215 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:22:49.256 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:22:49.299 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:22:49.313 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:22:49.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:49.355 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:49.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:49.457 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:49.495 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:49.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:49.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:49.642 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[29608]: pgmap v136: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 1.02k op/s 2024-03-24T11:22:49.642 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:22:49.642 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[29608]: 7.4 scrub starts 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/438668372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[37907]: pgmap v136: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 1.02k op/s 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[37907]: 7.4 scrub starts 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/438668372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:22:49.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:49 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:22:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:49.649 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:49.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:49 smithi181 ceph-mon[33998]: pgmap v136: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 1.02k op/s 2024-03-24T11:22:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:49 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:22:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:49 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:22:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:49 smithi181 ceph-mon[33998]: 7.4 scrub starts 2024-03-24T11:22:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:49 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/438668372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:22:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:49 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:22:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:49 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:22:49.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:49.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:49.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:49.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:49.923 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:49.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:50.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:50.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:50.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:50.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:50.187 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:50.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:50.288 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:22:50.288 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:22:50.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:50.346 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:50.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:50.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:22:50.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:50.501 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:50.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:50.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:50.620 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:22:50.620 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:22:50.620 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[29608]: 7.5 scrub starts 2024-03-24T11:22:50.620 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:22:50.620 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:22:50.621 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[29608]: 7.0 scrub starts 2024-03-24T11:22:50.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:22:50.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:22:50.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[37907]: 7.5 scrub starts 2024-03-24T11:22:50.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:22:50.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:22:50.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:50 smithi179 ceph-mon[37907]: 7.0 scrub starts 2024-03-24T11:22:50.621 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:22:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:22:50.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:50.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:50.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:50 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:22:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:50 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:22:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:50 smithi181 ceph-mon[33998]: 7.5 scrub starts 2024-03-24T11:22:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:50 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:22:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:50 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:22:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:50 smithi181 ceph-mon[33998]: 7.0 scrub starts 2024-03-24T11:22:50.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:50.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:50.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:50.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:50.911 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:50.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:51.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:51.054 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:51.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:51.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:51.183 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:51.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:51.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:51.322 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:51.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:51.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:51.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:51.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:51.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:51.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:51.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:51.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[29608]: 3.2 scrub starts 2024-03-24T11:22:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[29608]: 3.2 scrub ok 2024-03-24T11:22:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[29608]: pgmap v137: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:22:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:22:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:22:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:22:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:22:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[37907]: 3.2 scrub starts 2024-03-24T11:22:51.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[37907]: 3.2 scrub ok 2024-03-24T11:22:51.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[37907]: pgmap v137: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:22:51.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:22:51.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:22:51.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:22:51.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:51 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:22:51.728 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:51.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:51.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:51.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:51.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:51.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:51 smithi181 ceph-mon[33998]: 3.2 scrub starts 2024-03-24T11:22:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:51 smithi181 ceph-mon[33998]: 3.2 scrub ok 2024-03-24T11:22:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:51 smithi181 ceph-mon[33998]: pgmap v137: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:22:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:51 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:22:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:51 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:22:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:51 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:22:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:51 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:22:51.960 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:22:51.969 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:22:51.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-24T11:22:52.028 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:52.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:52.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:52.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:52.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:52.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:52.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:52.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:52.355 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:22:52.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:52.450 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:52.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:52.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:52.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:52.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:52.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[37907]: 7.5 scrub ok 2024-03-24T11:22:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[37907]: 7.4 scrub ok 2024-03-24T11:22:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:22:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:22:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:22:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:22:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[37907]: 7.0 scrub ok 2024-03-24T11:22:52.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[29608]: 7.5 scrub ok 2024-03-24T11:22:52.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[29608]: 7.4 scrub ok 2024-03-24T11:22:52.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:22:52.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:22:52.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:22:52.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:22:52.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:52 smithi179 ceph-mon[29608]: 7.0 scrub ok 2024-03-24T11:22:52.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:52.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:52.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:52.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:22:52.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:52 smithi181 ceph-mon[33998]: 7.5 scrub ok 2024-03-24T11:22:52.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:52 smithi181 ceph-mon[33998]: 7.4 scrub ok 2024-03-24T11:22:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:52 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:22:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:52 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:22:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:52 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:22:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:52 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:22:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:52 smithi181 ceph-mon[33998]: 7.0 scrub ok 2024-03-24T11:22:52.935 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:52.936 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:52.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:52.994 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:53.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:53.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:53.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:53.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:53.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:53.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:53.294 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:53.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:53.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:53.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:53.488 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:53.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:53.571 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:53.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:53.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:53.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:53.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:53 smithi181 ceph-mon[33998]: pgmap v138: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:22:53.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:53 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:22:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:53 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:22:53.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:22:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:22:53.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:53 smithi179 ceph-mon[37907]: pgmap v138: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:22:53.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:53 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:22:53.688 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:53 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:22:53.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:53 smithi179 ceph-mon[29608]: pgmap v138: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:22:53.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:53 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:22:53.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:53 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:22:53.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:53.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:53.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:53.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:53.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:53.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:53.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:54.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:54.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:54.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:54.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:54.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:54.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:54.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:54.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:54.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:54.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:22:54.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:22:54.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:54.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:22:54.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:22:54.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:22:54.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:54 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:22:54.633 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:54.691 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:54.793 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:54.833 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:54 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:22:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:54 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:22:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:22:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:54 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:22:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:54 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:22:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:54 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:22:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:54 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:22:54.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:54.979 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:54.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:55.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:55.121 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:55.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:55.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:55.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:55.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:55.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:55.371 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:22:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:55.399 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:55.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:55.501 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:55.545 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:55.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:55.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:55.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:55.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[29608]: pgmap v139: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s 2024-03-24T11:22:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:22:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:22:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:22:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:22:55.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[37907]: pgmap v139: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s 2024-03-24T11:22:55.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:22:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:22:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:22:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:55 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:22:55.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:55.835 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:55.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:55.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:55 smithi181 ceph-mon[33998]: pgmap v139: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s 2024-03-24T11:22:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:55 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:22:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:55 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:22:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:55 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:22:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:55 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:22:55.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:55.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:55.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:56.066 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:56.102 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:56.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:56.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:22:56.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:56.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:22:56.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:56.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:56.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:56.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:56.525 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:56.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:56.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:56.656 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:56.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:22:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:22:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:22:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:22:56.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:22:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:22:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:22:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:56 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:22:56.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:56.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:56.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:56.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:56.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:56 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:22:56.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:56 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:22:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:56 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:22:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:56 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:22:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:56.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:56.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:57.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:57.073 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:57.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:57.175 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:57.218 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:57.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:57.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:57.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:57.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:57.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:57.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:57.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:57.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:57.630 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:57.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:57.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[29608]: 3.7 scrub starts 2024-03-24T11:22:57.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[29608]: 3.7 scrub ok 2024-03-24T11:22:57.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[29608]: pgmap v140: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 808 op/s 2024-03-24T11:22:57.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:22:57.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:22:57.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[37907]: 3.7 scrub starts 2024-03-24T11:22:57.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[37907]: 3.7 scrub ok 2024-03-24T11:22:57.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[37907]: pgmap v140: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 808 op/s 2024-03-24T11:22:57.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:22:57.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:57 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:22:57.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:57.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:57.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:57.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:57.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:57.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:57.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:57 smithi181 ceph-mon[33998]: 3.7 scrub starts 2024-03-24T11:22:57.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:57 smithi181 ceph-mon[33998]: 3.7 scrub ok 2024-03-24T11:22:57.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:57 smithi181 ceph-mon[33998]: pgmap v140: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 808 op/s 2024-03-24T11:22:57.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:57 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:22:57.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:57 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:22:57.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:58.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:58.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:22:58.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:22:58.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:58.285 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:58.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:58.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:58.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:22:58.560 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:22:58.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.704 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:58.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:22:58.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:58.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:58.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:58.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:58.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:59.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:59.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:59.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:59.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:22:59.227 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:59.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:22:59.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:22:59.358 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:59.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:22:59.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:22:59.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:59.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:22:59.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:59.618 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:59.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[29608]: pgmap v141: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 846 op/s 2024-03-24T11:22:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:22:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:22:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:22:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:22:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:22:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:22:59.710 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[37907]: pgmap v141: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 846 op/s 2024-03-24T11:22:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:22:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:22:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:22:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:22:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:22:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:22:59 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:22:59.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:22:59.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:22:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:59.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:22:59.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:22:59.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:59 smithi181 ceph-mon[33998]: pgmap v141: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 846 op/s 2024-03-24T11:22:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:59 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:22:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:59 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:22:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:59 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:22:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:59 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:22:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:59 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:22:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:22:59 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:22:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:22:59.905 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:22:59.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:00.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:00.064 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:00.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:00.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:00.217 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:00.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:00.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:00.345 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:00.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:00.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:00.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:00.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:00.490 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:23:00.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:00.602 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:00.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:00.612 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 4.10 scrub starts 2024-03-24T11:23:00.612 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 4.10 scrub ok 2024-03-24T11:23:00.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:23:00.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:23:00.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:23:00.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:23:00.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:23:00.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:23:00.614 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2852239646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:00.614 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 4.10 scrub starts 2024-03-24T11:23:00.614 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 4.10 scrub ok 2024-03-24T11:23:00.614 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:23:00.615 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:23:00.615 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:23:00.615 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:23:00.615 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:23:00.615 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:23:00.615 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2852239646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:00.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:00.720 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:23:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:00.743 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:00.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:00.770 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:23:00.805 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:23:00.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:00.845 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:23:00.884 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:23:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:00.887 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:00.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 4.10 scrub starts 2024-03-24T11:23:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 4.10 scrub ok 2024-03-24T11:23:00.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:23:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:23:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:23:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:23:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:23:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:23:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2852239646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:00.921 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:23:00.965 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:23:00.988 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:01.009 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:23:01.015 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:23:01.027 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:01.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:01.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:01.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:01.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:01.304 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:01.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:01.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:01.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:01.495 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:01.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:01.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:01.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:01.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[29608]: pgmap v142: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:23:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:23:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:23:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2852239646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[29608]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:23:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[29608]: 7.6 deep-scrub starts 2024-03-24T11:23:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/336632130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[37907]: pgmap v142: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:23:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:23:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:23:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2852239646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[37907]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:23:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[37907]: 7.6 deep-scrub starts 2024-03-24T11:23:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/336632130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:01.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:01.765 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:01.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:01.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:01 smithi181 ceph-mon[33998]: pgmap v142: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:23:01.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:01 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:23:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:01 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:23:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2852239646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:01 smithi181 ceph-mon[33998]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T11:23:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:01 smithi181 ceph-mon[33998]: 7.6 deep-scrub starts 2024-03-24T11:23:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/336632130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:01.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:01.894 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:01.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:01.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:02.032 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:23:02.104 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:23:02.105 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:23:02.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:02.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:02.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:02.307 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:02.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:02.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:02.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:02.569 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:02.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:23:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:23:02.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:23:02.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:23:02.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:23:02.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:23:02.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:02.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:02 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:02.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.713 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:02.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:23:02.795 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:23:02.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-24T11:23:02.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.841 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:02.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:02.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:02 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:02.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:02.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:02.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:03.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:03.099 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:03.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:03.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:03.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:03.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:03.355 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:03.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:23:03.392 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:03.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:03.493 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:23:03.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:03.542 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:03.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:03.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: pgmap v145: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 671 op/s 2024-03-24T11:23:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:23:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:23:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 5.17 scrub starts 2024-03-24T11:23:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 5.17 scrub ok 2024-03-24T11:23:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:23:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:23:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:23:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: pgmap v145: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 671 op/s 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 5.17 scrub starts 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 5.17 scrub ok 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:23:03.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:03 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:23:03.744 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:03.747 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:03.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:03.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:03.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:03.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: pgmap v145: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 671 op/s 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 5.17 scrub starts 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 5.17 scrub ok 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:23:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:03 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:23:03.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:04.011 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:04.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:04.138 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:04.240 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:04.277 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:04.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:04.417 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:04.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:04.545 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:04.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:04.676 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:04 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:23:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:04 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:23:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:04 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:23:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:04 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:23:04.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:04.833 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:04 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:23:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:04 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:23:04.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:04.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:04.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:04.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:05.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:05.096 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:05.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:05.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:05.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:05.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:05.230 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:23:05.263 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:05.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:05.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:05.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:05.393 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:23:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:23:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:23:05.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:05.513 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:05.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:05.608 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:23:05.608 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:23:05.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:23:05.608 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:23:05.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.608 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26035 7.7 e99/92 7:e022efb2:::benchmark_data_smithi179_78267_object26034:head v 99'1561, mlcod=99'1556)", 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:17.814105+0000", 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.79128372300001, 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.110324294, 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.814105+0000", 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.609 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.814105+0000", 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.814106+0000", 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.75e-07 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.814167+0000", 2024-03-24T11:23:05.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1363000000000002e-05 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.814177+0000", 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0210000000000001e-05 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.814184+0000", 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.584e-06 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.816967+0000", 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027834460000000002 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.816985+0000", 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7805999999999999e-05 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819566+0000", 2024-03-24T11:23:05.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025810799999999999 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.924429+0000", 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10486323 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26043 7.3 7:c926c9c7:::benchmark_data_smithi179_78267_object26042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:17.815454+0000", 2024-03-24T11:23:05.613 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.78993432999999, 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.109794717, 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26043 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815454+0000", 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815454+0000", 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815456+0000", 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.409e-06 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815483+0000", 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6995000000000002e-05 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815493+0000", 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0711e-05 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815502+0000", 2024-03-24T11:23:05.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1699999999999997e-06 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815554+0000", 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1882999999999998e-05 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.815660+0000", 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106273 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:23:05.617 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.816414+0000", 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00075390500000000003 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818435+0000", 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020213169999999999 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819303+0000", 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00086838200000000001 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819470+0000", 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016618700000000001 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819634+0000", 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000164117 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.925249+0000", 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.105615368 2024-03-24T11:23:05.619 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26036 7.f e99/92 7:f83980d6:::benchmark_data_smithi179_78267_object26035:head v 99'1631, mlcod=99'1627)", 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:17.818376+0000", 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.78701284500001, 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10762295700000001, 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818376+0000", 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818376+0000", 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818377+0000", 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0999999999999998e-07 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818405+0000", 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8941999999999998e-05 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818416+0000", 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0365e-05 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818422+0000", 2024-03-24T11:23:05.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.421e-06 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818828+0000", 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040602599999999998 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.818847+0000", 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8609000000000001e-05 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.623 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.925914+0000", 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.1070667 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.925999+0000", 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5184000000000003e-05 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26051 7.3 7:c04930a3:::benchmark_data_smithi179_78267_object26050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:17.819798+0000", 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.785590945, 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.109994337, 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26051 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819798+0000", 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819798+0000", 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819799+0000", 2024-03-24T11:23:05.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2079999999999999e-06 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.819821+0000", 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1783000000000001e-05 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.924352+0000", 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10453144 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.924392+0000", 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0297999999999999e-05 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.925291+0000", 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00089901800000000002 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.925411+0000", 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119134 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.925532+0000", 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012120000000000001 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.929030+0000", 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003497942 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.929083+0000", 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3675999999999999e-05 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.929686+0000", 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00060250699999999996 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.630 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.929733+0000", 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7023e-05 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:17.929792+0000", 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9107999999999997e-05 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.631 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26479 7.a 7:5f28562f:::benchmark_data_smithi179_78267_object26478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.48758608, 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15488612299999999, 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26479 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2200000000000003e-07 2024-03-24T11:23:05.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117824+0000", 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0343000000000002e-05 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117833+0000", 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7610000000000002e-06 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117840+0000", 2024-03-24T11:23:05.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0350000000000004e-06 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121526+0000", 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003685912 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121658+0000", 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013200899999999999 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121793+0000", 2024-03-24T11:23:05.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134466 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272497+0000", 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15070414700000001 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272537+0000", 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9535e-05 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272575+0000", 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8544999999999998e-05 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272634+0000", 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9206999999999997e-05 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272689+0000", 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4441000000000003e-05 2024-03-24T11:23:05.637 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26483 7.2 7:427617cd:::benchmark_data_smithi179_78267_object26482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.486721468, 2024-03-24T11:23:05.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.160267666, 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26483 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118668+0000", 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.221e-06 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118691+0000", 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2310000000000002e-05 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118703+0000", 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2140999999999999e-05 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118711+0000", 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6739999999999997e-06 2024-03-24T11:23:05.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122939+0000", 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042282209999999999 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123051+0000", 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011178199999999999 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:23:05.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123184+0000", 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013356800000000001 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278443+0000", 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155258693 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278527+0000", 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4247999999999997e-05 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278810+0000", 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028318000000000001 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278891+0000", 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1002999999999997e-05 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278935+0000", 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3625000000000003e-05 2024-03-24T11:23:05.645 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26482 7.6 e99/92 7:67386a8b:::benchmark_data_smithi179_78267_object26481:head v 99'1627, mlcod=99'1625)", 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.483320866, 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15002405799999999, 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.646 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122069+0000", 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.057e-06 2024-03-24T11:23:05.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122090+0000", 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1392000000000001e-05 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122100+0000", 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9899999999999992e-06 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122107+0000", 2024-03-24T11:23:05.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0010000000000004e-06 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122268+0000", 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016068699999999999 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122287+0000", 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9508999999999999e-05 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272051+0000", 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.149763221 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272092+0000", 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1201000000000001e-05 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.650 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26485 7.6 e99/92 7:687a6aed:::benchmark_data_smithi179_78267_object26484:head v 99'1628, mlcod=99'1625)", 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.483262905, 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15087070799999999, 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8000000000000001e-07 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122148+0000", 2024-03-24T11:23:05.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1658000000000002e-05 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122155+0000", 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8600000000000004e-06 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122160+0000", 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.288e-06 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270971+0000", 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14881098400000001 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270992+0000", 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1002e-05 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272958+0000", 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019664079999999998 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272996+0000", 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8127999999999998e-05 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26488 7.a 7:57aba21a:::benchmark_data_smithi179_78267_object26487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.483033683, 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.156446115, 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.655 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26488 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122356+0000", 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.153e-06 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122473+0000", 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011658399999999999 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122488+0000", 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5500999999999999e-05 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122496+0000", 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1389999999999995e-06 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123136+0000", 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00064008700000000001 2024-03-24T11:23:05.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271318+0000", 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14818203499999999 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271479+0000", 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000160612 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.273137+0000", 2024-03-24T11:23:05.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016574700000000001 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278568+0000", 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054316099999999999 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278633+0000", 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4759999999999997e-05 2024-03-24T11:23:05.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278755+0000", 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012220500000000001 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278801+0000", 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5958999999999998e-05 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26489 7.3 7:c26b0ca7:::benchmark_data_smithi179_78267_object26488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.482876272, 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15559326900000001, 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.662 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26489 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122513+0000", 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.44e-07 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122535+0000", 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2070999999999999e-05 2024-03-24T11:23:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122543+0000", 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.763e-06 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122549+0000", 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0519999999999998e-06 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123004+0000", 2024-03-24T11:23:05.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045522100000000001 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123123+0000", 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119145 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271099+0000", 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14797587000000001 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.275853+0000", 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047536339999999996 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.276560+0000", 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00070673899999999996 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278000+0000", 2024-03-24T11:23:05.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014403090000000001 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278059+0000", 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9514e-05 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278106+0000", 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6307000000000002e-05 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.668 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26490 7.1 e99/92 7:8f3a7cc4:::benchmark_data_smithi179_78267_object26489:head v 99'1661, mlcod=99'1659)", 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.482357182, 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14983597100000001, 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:23:05.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123032+0000", 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5700000000000002e-07 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123062+0000", 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.006e-05 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271241+0000", 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14817881199999999 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271247+0000", 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3219999999999998e-06 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271300+0000", 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3066000000000003e-05 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271320+0000", 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0044e-05 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272827+0000", 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015071290000000001 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272867+0000", 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9981000000000002e-05 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26493 7.3 7:cc4c69e2:::benchmark_data_smithi179_78267_object26492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.482208566, 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155262276, 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.673 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26493 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123181+0000", 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9100000000000002e-07 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123215+0000", 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3609000000000003e-05 2024-03-24T11:23:05.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270946+0000", 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14773181099999999 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270967+0000", 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0211e-05 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274532+0000", 2024-03-24T11:23:05.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035658159999999999 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274642+0000", 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010941 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274767+0000", 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012473399999999999 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.277988+0000", 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032216419999999998 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278115+0000", 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012678200000000001 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278340+0000", 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000224589 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278405+0000", 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5335999999999999e-05 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278442+0000", 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7444999999999998e-05 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:26479 7.a e99/92 ondisk, result = 0)", 2024-03-24T11:23:05.679 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 107.481108992, 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14841649300000001, 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:23:05.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4300000000000003e-07 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124285+0000", 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8049999999999997e-06 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271371+0000", 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14708589 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271380+0000", 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9910000000000008e-06 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272564+0000", 2024-03-24T11:23:05.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011840279999999999 2024-03-24T11:23:05.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:05 smithi179 ceph-mon[29608]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:23:05.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:05 smithi179 ceph-mon[37907]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:23:05.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272570+0000", 2024-03-24T11:23:05.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7880000000000003e-06 2024-03-24T11:23:05.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272696+0000", 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126348 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:31032 7.2 7:4aa29d2d:::benchmark_data_smithi179_78267_object31031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:20.817020+0000", 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 104.7883688, 2024-03-24T11:23:05.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.121642214, 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 31032 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.817020+0000", 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.817020+0000", 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.817021+0000", 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1800000000000004e-07 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.817042+0000", 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1169999999999999e-05 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.817053+0000", 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1131000000000001e-05 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.817061+0000", 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5340000000000002e-06 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.818818+0000", 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001757558 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.819035+0000", 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021731499999999999 2024-03-24T11:23:05.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.819145+0000", 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109215 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.890984+0000", 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.071839582999999999 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.891018+0000", 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3763000000000002e-05 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.938532+0000", 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.047514088000000003 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.938608+0000", 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6119999999999996e-05 2024-03-24T11:23:05.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:20.938662+0000", 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3819000000000001e-05 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:126613 7.f e118/92 7:fdd7a776:::benchmark_data_smithi179_78267_object7419:head v 118'7962, mlcod=118'7960)", 2024-03-24T11:23:05.692 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:44.573091+0000", 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.032297918000001, 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.114612744, 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.573091+0000", 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.573091+0000", 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.573091+0000", 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0900000000000001e-07 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.573098+0000", 2024-03-24T11:23:05.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.798e-06 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.573108+0000", 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5429999999999999e-06 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.573115+0000", 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1149999999999996e-06 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.575134+0000", 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020194179999999998 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.575159+0000", 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4425000000000001e-05 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.687665+0000", 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11250584600000001 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.687703+0000", 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8890000000000002e-05 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.697 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:126624 7.3 7:c0442e72:::benchmark_data_smithi179_78267_object7430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:44.576257+0000", 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.029132002000001, 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10818198900000001, 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 126624 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576257+0000", 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576257+0000", 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576258+0000", 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5090000000000001e-06 2024-03-24T11:23:05.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576265+0000", 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7100000000000001e-06 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576279+0000", 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3949e-05 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576288+0000", 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8289999999999997e-06 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576347+0000", 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9478999999999998e-05 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576474+0000", 2024-03-24T11:23:05.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127145 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576607+0000", 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0001325 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.580781+0000", 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041744920000000001 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.702 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.580819+0000", 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7920999999999998e-05 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684228+0000", 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.103408982 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684388+0000", 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016014100000000001 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684439+0000", 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0331999999999999e-05 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:126625 7.1 e118/92 7:8279287a:::benchmark_data_smithi179_78267_object7431:head v 118'7885, mlcod=118'7883)", 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:44.576617+0000", 2024-03-24T11:23:05.704 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.028771271, 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10794266700000001, 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576617+0000", 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576617+0000", 2024-03-24T11:23:05.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576618+0000", 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5700000000000002e-07 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576627+0000", 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1689999999999998e-06 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576638+0000", 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0655e-05 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576644+0000", 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5200000000000003e-06 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576726+0000", 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1843999999999998e-05 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.576744+0000", 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7856999999999999e-05 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684515+0000", 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.107770426 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684560+0000", 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5439000000000003e-05 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:126629 7.1 e118/92 7:86ceb58f:::benchmark_data_smithi179_78267_object7435:head v 118'7886, mlcod=118'7883)", 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:44.581208+0000", 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.024180210000001, 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11152585199999999, 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.581208+0000", 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.709 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.581208+0000", 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.581209+0000", 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0100000000000005e-07 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.710 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.581215+0000", 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8619999999999999e-06 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684173+0000", 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.102958329 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684607+0000", 2024-03-24T11:23:05.711 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000433499 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684648+0000", 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1627e-05 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684667+0000", 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8913e-05 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.712 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.692693+0000", 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0080257249999999992 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.692734+0000", 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1295999999999999e-05 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:126634 7.a 7:50a82d1a:::benchmark_data_smithi179_78267_object7440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:23:05.713 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:44.582264+0000", 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.023124578000001, 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11651969, 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 126634 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.714 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.582264+0000", 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.582264+0000", 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.582266+0000", 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.669e-06 2024-03-24T11:23:05.715 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.716 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.716 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.716 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.582271+0000", 2024-03-24T11:23:05.716 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3600000000000004e-06 2024-03-24T11:23:05.716 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.716 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.716 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.717 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684198+0000", 2024-03-24T11:23:05.717 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.101926667 2024-03-24T11:23:05.717 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.717 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.717 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684227+0000", 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9689e-05 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684598+0000", 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00037082499999999997 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.686183+0000", 2024-03-24T11:23:05.718 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001584614 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.686288+0000", 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000105462 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.688749+0000", 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002460612 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.719 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.691617+0000", 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028682640000000001 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.698691+0000", 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070736970000000003 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.698743+0000", 2024-03-24T11:23:05.720 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2259999999999998e-05 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.698784+0000", 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0571000000000002e-05 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:126631 7.f e118/92 7:ff561a6f:::benchmark_data_smithi179_78267_object7437:head v 118'7964, mlcod=118'7960)", 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:44.586541+0000", 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 21.018847887, 2024-03-24T11:23:05.721 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11122952699999999, 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.586541+0000", 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.586541+0000", 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:23:05.722 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.586542+0000", 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.11e-06 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.586551+0000", 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4919999999999999e-06 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684191+0000", 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.097639129000000005 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684217+0000", 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6347e-05 2024-03-24T11:23:05.724 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684544+0000", 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00032696500000000002 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.684566+0000", 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2385e-05 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.697731+0000", 2024-03-24T11:23:05.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013164853000000001 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:44.697770+0000", 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9246000000000003e-05 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:23:05.726 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:23:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:05.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:05.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:05.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:05.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:05 smithi181 ceph-mon[33998]: pgmap v146: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:23:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:05.903 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:05.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:06.004 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:06.040 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:06.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:06.181 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:06.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:06.314 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:06.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:06.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.454 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:06.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:06.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.594 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:06.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:23:06.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:06 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:23:06.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:06.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:06.822 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:06.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:06 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:23:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:06 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:23:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:06 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:23:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:06 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:23:06.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:06.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:06.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:07.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:23:07.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:07.128 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:07.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:07.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:07.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:07.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:07.364 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:07.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:07.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:07.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:07.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:07.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:07.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:07.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:07.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:07.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:07.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: pgmap v147: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 4.1f deep-scrub starts 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 4.1f deep-scrub ok 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 3.b scrub starts 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 3.b scrub ok 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:23:07.797 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:23:07.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:07 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:23:07.800 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:07.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:07.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:07.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: pgmap v147: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 4.1f deep-scrub starts 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 4.1f deep-scrub ok 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 3.b scrub starts 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 3.b scrub ok 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:23:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: pgmap v147: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 4.1f deep-scrub starts 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 4.1f deep-scrub ok 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 3.b scrub starts 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 3.b scrub ok 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:23:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:07 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:23:08.031 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:08.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:08.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:08.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:08.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:08.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:23:08.278 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:23:08.279 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:23:08.279 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:23:08.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:08.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:08.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:08.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:08.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:08.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:08.579 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:08.610 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:08.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:08.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:08.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:08.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:08.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:08.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:08 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:23:08.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:08 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:23:08.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:08 smithi181 ceph-mon[33998]: 6.2 scrub starts 2024-03-24T11:23:08.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:08 smithi181 ceph-mon[33998]: 6.2 scrub ok 2024-03-24T11:23:08.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:08 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:23:08.906 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:08.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:23:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:23:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[29608]: 6.2 scrub starts 2024-03-24T11:23:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[29608]: 6.2 scrub ok 2024-03-24T11:23:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:23:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:23:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:23:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[37907]: 6.2 scrub starts 2024-03-24T11:23:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[37907]: 6.2 scrub ok 2024-03-24T11:23:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:08 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:23:09.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:23:09.011 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:23:09.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-24T11:23:09.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:09.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:09.138 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:09.185 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:09.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:09.286 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:09.337 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:09.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:09.361 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:23:09.361 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:23:09.361 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-03-24T11:23:09.361 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:23:09.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:09.464 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:09.529 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:09.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:09.552 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:09.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:09 smithi179 ceph-mon[29608]: pgmap v148: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 654 op/s 2024-03-24T11:23:09.614 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:09 smithi179 ceph-mon[37907]: pgmap v148: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 654 op/s 2024-03-24T11:23:09.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:09.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:09.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:09.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:09.800 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:09.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:09.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:09 smithi181 ceph-mon[33998]: pgmap v148: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 654 op/s 2024-03-24T11:23:09.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:09.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:09.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:10.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:10.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:10.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:10.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:23:10.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:10.206 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:10.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:10.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:10.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:10.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:10.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:10.467 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:10.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:10.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:10.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:10.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:10.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:10.743 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:10.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:10.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:10.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:23:10.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[37907]: 6.c scrub starts 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[37907]: 6.c scrub ok 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[29608]: 6.c scrub starts 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[29608]: 6.c scrub ok 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:23:10.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:10 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:23:10.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:10 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:23:10.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:10 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:23:10.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:10 smithi181 ceph-mon[33998]: 6.c scrub starts 2024-03-24T11:23:10.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:10 smithi181 ceph-mon[33998]: 6.c scrub ok 2024-03-24T11:23:10.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:10 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:23:10.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:10 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:23:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:10.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:10.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:10.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:11.015 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:11.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:11.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:11.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:11.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:11.380 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:11.417 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:11.441 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:11.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:11.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:11.568 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:23:11.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:11.688 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:11.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:11.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:11.842 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:11 smithi181 ceph-mon[33998]: pgmap v149: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 918 op/s 2024-03-24T11:23:11.842 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:11 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:23:11.842 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:11 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:23:11.842 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:11 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2239126062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:11.842 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:11 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:23:11.842 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:11 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:23:11.905 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[37907]: pgmap v149: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 918 op/s 2024-03-24T11:23:11.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:23:11.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:23:11.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2239126062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:11.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:23:11.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:23:11.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[29608]: pgmap v149: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 918 op/s 2024-03-24T11:23:11.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:23:11.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:23:11.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2239126062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:11.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:23:11.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:11 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:23:11.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:11.960 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:11.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:12.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:12.091 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:12.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:12.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:12.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:12.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:12.328 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:12.371 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:12.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:12.473 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:12.511 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:12.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:12.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:12.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:12.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:12.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:12.787 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:12.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:12.834 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:12 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:12.834 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2239126062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:23:12.834 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:12 smithi179 ceph-mon[29608]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:23:12.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:12 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:12.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:12 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2239126062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:23:12.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:12 smithi179 ceph-mon[37907]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:23:12.889 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:12 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:12 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2239126062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:23:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:12 smithi181 ceph-mon[33998]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T11:23:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:12.926 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:12.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:13.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:13.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:13.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:13.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:13.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:13.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:13.292 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:13.331 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:13.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:13.390 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:23:13.391 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:23:13.391 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:23:13.391 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:23:13.391 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:23:13.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:13.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:13.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:13.569 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:23:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:23:13.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:13.616 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:13.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:13.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:13.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:13.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:13.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:13.852 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:13 smithi181 ceph-mon[33998]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 881 op/s 2024-03-24T11:23:13.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:13.864 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:13 smithi179 ceph-mon[29608]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 881 op/s 2024-03-24T11:23:13.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:13 smithi179 ceph-mon[37907]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 881 op/s 2024-03-24T11:23:13.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:13.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:13.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:14.015 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:14.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:14.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:14.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.281 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:14.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:14.382 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:14.417 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:14.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:14.554 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:14.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:23:14.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:23:14.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:14.711 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:14.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:14.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:14.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:14.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:14.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:14.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:14.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:14.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:15.016 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:15.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:15.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:15.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:15.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:15.251 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:23:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:23:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:23:15.259 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:15.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:15.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:15.385 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:15.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:15.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:15.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:15.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:15.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:15.647 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:23:15.648 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:23:15.649 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:23:15.649 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:23:15.649 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:23:15.649 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:23:15.649 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:23:15.668 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:15.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:15.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:15.808 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:15.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:15.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:15 smithi181 ceph-mon[33998]: pgmap v152: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s 2024-03-24T11:23:15.855 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:15 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3128524896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:15.909 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:15.948 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:15.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:15 smithi179 ceph-mon[29608]: pgmap v152: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s 2024-03-24T11:23:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:15 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3128524896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:15 smithi179 ceph-mon[37907]: pgmap v152: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s 2024-03-24T11:23:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:15 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3128524896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:16.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:16.081 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:16.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:16.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:16.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:16.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:16.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:16.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:16.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:16.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:16.488 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:16.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:16.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:16.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:16.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:16.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:16.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:16.769 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:16.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:16.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:16.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:16.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:16.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:17.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:17.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:17.167 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:17.269 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:17.306 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:17.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:17.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:17.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:23:17.563 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:23:17.578 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:17.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:17.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:17.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:17.851 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:17 smithi181 ceph-mon[33998]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s 2024-03-24T11:23:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:23:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:23:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:17 smithi181 ceph-mon[33998]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:23:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:17 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:17 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:17.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[29608]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s 2024-03-24T11:23:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:23:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:23:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[29608]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:23:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[37907]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s 2024-03-24T11:23:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:23:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:23:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[37907]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T11:23:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:17 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:17.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:17.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:18.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:18.130 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:18.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:18.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:18.269 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:18.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:18.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:18.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:18.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:23:18.503 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:23:18.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-24T11:23:18.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:18.515 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:18.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:18.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:18.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:23:18.578 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:23:18.578 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:23:18.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:18.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:18.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:18.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:18.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:18.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:18.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:18.939 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:18.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:19.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:19.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:19.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:19.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:19.207 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:19.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:19.308 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:19.350 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:19.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:19.451 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:19.503 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:19.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:19.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:19.638 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:19.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:19.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:19.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:19.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:19.799 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:19 smithi179 ceph-mon[29608]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s 2024-03-24T11:23:19.799 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:19.800 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:19.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:19 smithi179 ceph-mon[37907]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s 2024-03-24T11:23:19.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:19.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:19.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:19.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:19 smithi181 ceph-mon[33998]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s 2024-03-24T11:23:19.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:19.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:19.917 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:19.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:19.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:19.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:19.962 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:20.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:23:20.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:20.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:20.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:20.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:20.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:20.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:20.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:20.360 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:20.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:20.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:20.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:20.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:20.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:20.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:20.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:20.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:20.782 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:20.827 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:20.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:20.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:20.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:20.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:21.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:21.100 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:21.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:21.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:21.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:21.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:21.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:21.361 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:21.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:21.463 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:21.500 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:21.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:21.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:21.622 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:23:21.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:21.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:21.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:21.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:21.790 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:21.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:21.848 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:21 smithi181 ceph-mon[33998]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 949 op/s 2024-03-24T11:23:21.848 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:21 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:23:21.848 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:21 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:23:21.848 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:21 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/617794367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:21.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:21.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:21.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[37907]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 949 op/s 2024-03-24T11:23:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:23:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:23:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/617794367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[29608]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 949 op/s 2024-03-24T11:23:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:23:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:23:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:21 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/617794367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:22.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:22.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:22.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:22.185 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:22.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.327 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:22.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:22.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:22.461 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:22.501 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:23:22.502 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:23:22.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:22.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:22.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:22.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:22.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:22 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:22.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:22 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/617794367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:22.822 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:22 smithi181 ceph-mon[33998]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:23:22.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:22.859 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:22 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:22 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/617794367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:22 smithi179 ceph-mon[29608]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:23:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:22 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:22 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/617794367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:22 smithi179 ceph-mon[37907]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T11:23:22.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:22.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:22.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:23.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:23.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:23.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:23.177 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:23:23.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:23.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:23.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:23.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:23.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:23.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:23.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:23:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:23:23.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:23.520 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:23.532 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:23.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:23.634 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:23.682 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:23.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:23 smithi179 ceph-mon[29608]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 496 op/s 2024-03-24T11:23:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:23 smithi179 ceph-mon[37907]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 496 op/s 2024-03-24T11:23:23.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:23.832 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:23.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:23 smithi181 ceph-mon[33998]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 496 op/s 2024-03-24T11:23:23.934 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:23.976 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:23.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:24.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:24.122 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:24.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:24.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:24.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:24.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:24.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:24.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:24.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:24.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:24.525 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:24.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:24.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:24.655 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:24.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:23:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:23:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[29608]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:23:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:23:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:23:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:24 smithi179 ceph-mon[37907]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:23:24.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:24.797 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:24.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:24.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:24.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:24.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:24 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:23:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:24 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:23:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:24 smithi181 ceph-mon[33998]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T11:23:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:24.940 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:24.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:25.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:25.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:25.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:25.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:25.228 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:25.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:25.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:25.352 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:23:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:23:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:23:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:25.380 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:25.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:25.419 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:23:25.420 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:23:25.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:25.530 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:25.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:25.633 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:25.668 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:25.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:25.773 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:25.812 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:25.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:25.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:25 smithi181 ceph-mon[33998]: pgmap v160: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:23:25.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:25.931 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:25 smithi179 ceph-mon[29608]: pgmap v160: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:23:25.932 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:25 smithi179 ceph-mon[37907]: pgmap v160: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:23:25.967 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:25.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:26.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:26.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:26.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:26.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:26.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:26.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:26.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:23:26.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:23:26.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:26.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:26.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:26.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:26.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:26.594 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:26.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:26.673 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:26.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:26.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:26.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:26.819 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:26.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:26.921 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:26.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:27.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:27.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:27.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:27.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:27.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:27.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:27.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:27.403 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:27.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:27.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:27.567 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:27.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:27.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:27.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:27.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:27.781 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:27 smithi179 ceph-mon[29608]: pgmap v161: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s 2024-03-24T11:23:27.781 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:27 smithi179 ceph-mon[37907]: pgmap v161: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s 2024-03-24T11:23:27.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:27.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:27.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:27.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:27 smithi181 ceph-mon[33998]: pgmap v161: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s 2024-03-24T11:23:27.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:27.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:27.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:28.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:28.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:28.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:23:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:23:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:23:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:23:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:23:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:23:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:23:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:23:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:23:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:23:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:23:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:23:28.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:28.227 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:23:28.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:28.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:28.269 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:23:28.318 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:23:28.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:28.374 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:28.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:28.380 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:23:28.421 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:23:28.460 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:23:28.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:28.509 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:23:28.553 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:23:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:28.558 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:28.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:23:28.571 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:23:28.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:28.688 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:28.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:28.737 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2349928085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:28.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:28.832 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:28.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:28.901 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:7858 7.0 7:0db796ce:::benchmark_data_smithi179_78267_object7857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:09.871668+0000", 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 139.02678368599999, 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.051397471, 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.902 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 7858 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871668+0000", 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871668+0000", 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871669+0000", 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3149999999999999e-06 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871729+0000", 2024-03-24T11:23:28.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9349999999999999e-05 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871741+0000", 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2303000000000001e-05 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871749+0000", 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2640000000000008e-06 2024-03-24T11:23:28.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871804+0000", 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5037999999999997e-05 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.871919+0000", 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114363 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.872084+0000", 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000165172 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.872658+0000", 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057421999999999996 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.873835+0000", 2024-03-24T11:23:28.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011768410000000001 2024-03-24T11:23:28.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.922964+0000", 2024-03-24T11:23:28.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049129008000000002 2024-03-24T11:23:28.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.923016+0000", 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1907000000000003e-05 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.923065+0000", 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.969e-05 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:7867 7.0 7:08ef7178:::benchmark_data_smithi179_78267_object7866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:09.874833+0000", 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 139.02361811099999, 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049563686000000003, 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.911 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 7867 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874833+0000", 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874833+0000", 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874835+0000", 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4139999999999999e-06 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874878+0000", 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3219000000000001e-05 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874892+0000", 2024-03-24T11:23:28.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4175999999999999e-05 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874900+0000", 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2620000000000001e-06 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874966+0000", 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6600000000000006e-05 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.875088+0000", 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121548 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.875237+0000", 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014949900000000001 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.876480+0000", 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001242677 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.876568+0000", 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8199e-05 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.924267+0000", 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047698764999999997 2024-03-24T11:23:28.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.924342+0000", 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5063999999999998e-05 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.924397+0000", 2024-03-24T11:23:28.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5263000000000003e-05 2024-03-24T11:23:28.918 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:7868 7.0 7:0ea8b2d9:::benchmark_data_smithi179_78267_object7867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:09.874918+0000", 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 139.02353322299999, 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049627771000000001, 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.919 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.920 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 7868 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874918+0000", 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874918+0000", 2024-03-24T11:23:28.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874919+0000", 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0399999999999996e-07 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874941+0000", 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1707000000000001e-05 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874948+0000", 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5140000000000004e-06 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.874954+0000", 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5269999999999998e-06 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.875707+0000", 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075287499999999996 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.875886+0000", 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017988999999999999 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.876075+0000", 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018901399999999999 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.876656+0000", 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058099099999999999 2024-03-24T11:23:28.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.877264+0000", 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00060763099999999997 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.924447+0000", 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047182492999999999 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.924505+0000", 2024-03-24T11:23:28.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8425999999999999e-05 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:09.924546+0000", 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1199000000000003e-05 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:21745 7.0 7:0df0b4a0:::benchmark_data_smithi179_78267_object21744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:15.729990+0000", 2024-03-24T11:23:28.926 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 133.168461304, 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067762227999999994, 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 21745 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.729990+0000", 2024-03-24T11:23:28.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.729990+0000", 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.729995+0000", 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5689999999999998e-06 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.730031+0000", 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6254000000000002e-05 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.730048+0000", 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7217000000000001e-05 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.730057+0000", 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8279999999999992e-06 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.730118+0000", 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0662999999999998e-05 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.730260+0000", 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014260600000000001 2024-03-24T11:23:28.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.730382+0000", 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121129 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731306+0000", 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00092471999999999997 2024-03-24T11:23:28.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731844+0000", 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00053766700000000003 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.797646+0000", 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.065801602000000001 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.797687+0000", 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1646000000000001e-05 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.797753+0000", 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5327000000000007e-05 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26498 7.0 7:0d5c2bb1:::benchmark_data_smithi179_78267_object26497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:28.933 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 130.77292500999999, 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14803461000000001, 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26498 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:23:28.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125528+0000", 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.584e-06 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125569+0000", 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0945000000000002e-05 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125586+0000", 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7198999999999999e-05 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125596+0000", 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4919999999999999e-06 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125670+0000", 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4246999999999996e-05 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125813+0000", 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014268000000000001 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126002+0000", 2024-03-24T11:23:28.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000189626 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126981+0000", 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00097871299999999998 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127606+0000", 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062507399999999999 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273447+0000", 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14584118800000001 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273512+0000", 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4628999999999994e-05 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273561+0000", 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9233000000000001e-05 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.939 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:31277 7.0 7:016005f4:::benchmark_data_smithi179_78267_object31276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:21.043957+0000", 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 127.854494258, 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.070952840000000003, 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.940 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 31277 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.043957+0000", 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.043957+0000", 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.043959+0000", 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.886e-06 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.043997+0000", 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7598000000000002e-05 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.044012+0000", 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5545000000000001e-05 2024-03-24T11:23:28.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.044020+0000", 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0020000000000006e-06 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.044084+0000", 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4086999999999998e-05 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.044223+0000", 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013815999999999999 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.044373+0000", 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015030100000000001 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.045573+0000", 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011996929999999999 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.046383+0000", 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081013999999999999 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.114828+0000", 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068445541999999998 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.114867+0000", 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9116000000000003e-05 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:21.114910+0000", 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2769999999999999e-05 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:65607 7.0 7:0abd0400:::benchmark_data_smithi179_78267_object65606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:44.076933+0000", 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 104.821518751, 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.071670922999999997, 2024-03-24T11:23:28.946 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 65607 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.076933+0000", 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.076933+0000", 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.076934+0000", 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1820000000000001e-06 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.076989+0000", 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4518999999999998e-05 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.076999+0000", 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0118999999999999e-05 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.077004+0000", 2024-03-24T11:23:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6710000000000004e-06 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.077051+0000", 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6542000000000003e-05 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.077156+0000", 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010508199999999999 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.077263+0000", 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010750400000000001 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.078963+0000", 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001700013 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.139851+0000", 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.060887575999999999 2024-03-24T11:23:28.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.148505+0000", 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0086543590000000004 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.148561+0000", 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5846e-05 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.148604+0000", 2024-03-24T11:23:28.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2509999999999998e-05 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:67447 7.0 7:07913d77:::benchmark_data_smithi179_78267_object67446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:45.793521+0000", 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 103.104930627, 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052859392999999998, 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.953 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 67447 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793521+0000", 2024-03-24T11:23:28.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793521+0000", 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793523+0000", 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.739e-06 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793576+0000", 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2954999999999997e-05 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793587+0000", 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1021e-05 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793592+0000", 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5480000000000001e-06 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793645+0000", 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2413000000000001e-05 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793780+0000", 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000135714 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.793921+0000", 2024-03-24T11:23:28.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014088699999999999 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.796173+0000", 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022522010000000001 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.802431+0000", 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006257383 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.846255+0000", 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043823793 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.846328+0000", 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3207999999999994e-05 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.846380+0000", 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2531000000000002e-05 2024-03-24T11:23:28.959 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:67459 7.0 7:0946cba0:::benchmark_data_smithi179_78267_object67458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:45.799915+0000", 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 103.098536821, 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057096053000000001, 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.960 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 67459 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.799915+0000", 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.799915+0000", 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.799916+0000", 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.186e-06 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.800042+0000", 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012643499999999999 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.800052+0000", 2024-03-24T11:23:28.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9000000000000001e-06 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.800060+0000", 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9179999999999994e-06 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.800123+0000", 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2502000000000006e-05 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.800255+0000", 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013256699999999999 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.800402+0000", 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014638499999999999 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.802508+0000", 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002106559 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.805654+0000", 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003145527 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.856900+0000", 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.051245937999999998 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.856960+0000", 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0714999999999997e-05 2024-03-24T11:23:28.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:45.857011+0000", 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0420999999999997e-05 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:92828 7.0 7:0662f45c:::benchmark_data_smithi179_78267_object92827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:09.778291+0000", 2024-03-24T11:23:28.966 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.120160741999996, 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066972216000000001, 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 92828 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778291+0000", 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778291+0000", 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778293+0000", 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0820000000000001e-06 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778417+0000", 2024-03-24T11:23:28.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000123645 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778435+0000", 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8814e-05 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778444+0000", 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0259999999999996e-06 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778507+0000", 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2717000000000003e-05 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778653+0000", 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000145451 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778810+0000", 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000157405 2024-03-24T11:23:28.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.781937+0000", 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003127023 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.782380+0000", 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044265 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.845146+0000", 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062766060999999998 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.845207+0000", 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1019e-05 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.845263+0000", 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6323000000000003e-05 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.972 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:94871 7.0 7:09452384:::benchmark_data_smithi179_78267_object94870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:11.832135+0000", 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 77.066316291999996, 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.065359570000000006, 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 94871 2024-03-24T11:23:28.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832135+0000", 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832135+0000", 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832145+0000", 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.73e-06 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832184+0000", 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9416999999999999e-05 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832201+0000", 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6167999999999998e-05 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832207+0000", 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.652e-06 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832259+0000", 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1353000000000001e-05 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832370+0000", 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011178100000000001 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.832487+0000", 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011661 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.834884+0000", 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023968570000000001 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.835652+0000", 2024-03-24T11:23:28.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076829399999999999 2024-03-24T11:23:28.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.897382+0000", 2024-03-24T11:23:28.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:28.979 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2349928085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:28.980 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2349928085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:28.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.061730107999999999 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.897449+0000", 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7144999999999995e-05 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:11.897495+0000", 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5454999999999998e-05 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.981 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:95764 7.0 7:0bda95a3:::benchmark_data_smithi179_78267_object95763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:12.884856+0000", 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 76.013595785999996, 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.058199019999999997, 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 95764 2024-03-24T11:23:28.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.884856+0000", 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.884856+0000", 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.884857+0000", 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5799999999999999e-06 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.884899+0000", 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1820000000000003e-05 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.884914+0000", 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4813e-05 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.884924+0000", 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7049999999999994e-06 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.884975+0000", 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1626999999999999e-05 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885072+0000", 2024-03-24T11:23:28.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6241000000000002e-05 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885218+0000", 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014649500000000001 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.892476+0000", 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0072579159999999997 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.900598+0000", 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081217679999999997 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.942938+0000", 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042339818000000001 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.943004+0000", 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6301000000000001e-05 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.943055+0000", 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0936000000000002e-05 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:95765 7.0 7:07913695:::benchmark_data_smithi179_78267_object95764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:12.885488+0000", 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 76.012964006999994, 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062479809999999997, 2024-03-24T11:23:28.988 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 95765 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885488+0000", 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885488+0000", 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885488+0000", 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1699999999999998e-07 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885560+0000", 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1799999999999997e-05 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885569+0000", 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9679999999999995e-06 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885575+0000", 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5400000000000003e-06 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.887783+0000", 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022080569999999998 2024-03-24T11:23:28.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.887890+0000", 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107828 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.888022+0000", 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000131376 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.892479+0000", 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044573340000000003 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.900637+0000", 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081580349999999992 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.947891+0000", 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047254065999999997 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.947932+0000", 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1278e-05 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.947967+0000", 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4910999999999998e-05 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:95767 7.0 7:0cbf28f0:::benchmark_data_smithi179_78267_object95766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-24T11:23:28.994 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:12.885871+0000", 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 76.012580549000006, 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062194887999999997, 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 95767 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:28.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885871+0000", 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885871+0000", 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885872+0000", 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7899999999999999e-07 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885953+0000", 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1927000000000004e-05 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885960+0000", 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5570000000000001e-06 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.885965+0000", 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4839999999999998e-06 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.890527+0000", 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045622070000000004 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.890614+0000", 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7361999999999995e-05 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.890709+0000", 2024-03-24T11:23:28.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4963999999999994e-05 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.892629+0000", 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001919855 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.900665+0000", 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0080356509999999996 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:28.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948008+0000", 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047343639999999999 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948042+0000", 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4034e-05 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948066+0000", 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3728000000000001e-05 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.000 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:95780 7.0 7:02e4dbed:::benchmark_data_smithi179_78267_object95779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:12.895227+0000", 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 76.003224583000005, 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052923368999999998, 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 95780 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895227+0000", 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895227+0000", 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895238+0000", 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0515e-05 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895274+0000", 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6035999999999999e-05 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895292+0000", 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8603000000000001e-05 2024-03-24T11:23:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895301+0000", 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4789999999999996e-06 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895356+0000", 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5575999999999998e-05 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895483+0000", 2024-03-24T11:23:29.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012658999999999999 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.895595+0000", 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011237400000000001 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.897704+0000", 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002108375 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.901804+0000", 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041006740000000003 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948102+0000", 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046297611000000002 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948131+0000", 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8648000000000001e-05 2024-03-24T11:23:29.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948150+0000", 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9888e-05 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:95781 7.0 7:0a038c6b:::benchmark_data_smithi179_78267_object95780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:12.898553+0000", 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 75.999898751000003, 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049670177000000003, 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:29.007 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 95781 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898553+0000", 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898553+0000", 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898554+0000", 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0580000000000001e-06 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898598+0000", 2024-03-24T11:23:29.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4329999999999997e-05 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898610+0000", 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1663e-05 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898617+0000", 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9469999999999997e-06 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898656+0000", 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9338999999999997e-05 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898760+0000", 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103461 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.898852+0000", 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2818000000000005e-05 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.900672+0000", 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001819677 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.901839+0000", 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011666160000000001 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948175+0000", 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046335926999999999 2024-03-24T11:23:29.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948203+0000", 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7854000000000001e-05 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:12.948223+0000", 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0486999999999999e-05 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126396 7.0 7:052b3342:::benchmark_data_smithi179_78267_object7202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.345705+0000", 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.552747033999999, 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.050258820000000003, 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126396 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.014 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345705+0000", 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345705+0000", 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345706+0000", 2024-03-24T11:23:29.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.618e-06 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345713+0000", 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0929999999999997e-06 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345738+0000", 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5055999999999999e-05 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345748+0000", 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3200000000000006e-06 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345803+0000", 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5216000000000001e-05 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345886+0000", 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3553000000000005e-05 2024-03-24T11:23:29.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.345958+0000", 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1876000000000001e-05 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.348234+0000", 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022755420000000002 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.349426+0000", 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001192533 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.395880+0000", 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.046453375999999998 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.395934+0000", 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3909000000000002e-05 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.395963+0000", 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9728000000000001e-05 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126626 7.0 7:0b9b7f0f:::benchmark_data_smithi179_78267_object7432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.322228500000001, 2024-03-24T11:23:29.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10890327499999999, 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126626 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576224+0000", 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.172e-06 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576229+0000", 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5279999999999997e-06 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576240+0000", 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0963e-05 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576248+0000", 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6589999999999996e-06 2024-03-24T11:23:29.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576302+0000", 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3278e-05 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576422+0000", 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119988 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:29.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576496+0000", 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4645000000000001e-05 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.579163+0000", 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002667123 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580732+0000", 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001568124 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685008+0000", 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10427672 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685071+0000", 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2853000000000003e-05 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685126+0000", 2024-03-24T11:23:29.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5222000000000001e-05 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126640 7.0 7:05a202e2:::benchmark_data_smithi179_78267_object7446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.307280274999997, 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097235629000000004, 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126640 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:29.028 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591173+0000", 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.327e-06 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591177+0000", 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0589999999999996e-06 2024-03-24T11:23:29.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591188+0000", 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0954e-05 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591194+0000", 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3400000000000003e-06 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:29.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591252+0000", 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7930999999999998e-05 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591368+0000", 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115586 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591497+0000", 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129834 2024-03-24T11:23:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.593983+0000", 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024858359999999999 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.594318+0000", 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033431900000000002 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688305+0000", 2024-03-24T11:23:29.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093987494000000005 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688361+0000", 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6203999999999999e-05 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688407+0000", 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5745000000000003e-05 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.033 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:154480 7.0 7:09166c18:::benchmark_data_smithi179_78267_object35286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.798776494, 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099165692999999999, 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 154480 2024-03-24T11:23:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099677+0000", 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.575e-06 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099684+0000", 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6120000000000001e-06 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099697+0000", 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2541000000000001e-05 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099704+0000", 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5499999999999997e-06 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099976+0000", 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000271122 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.100084+0000", 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010849900000000001 2024-03-24T11:23:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.100211+0000", 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012710099999999999 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.103105+0000", 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028942939999999999 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106238+0000", 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003132953 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198758+0000", 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092520054000000004 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198797+0000", 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8912000000000003e-05 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198841+0000", 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3479999999999997e-05 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:29.040 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:23:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:23:29.042 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:23:29.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-24T11:23:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:29.043 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:29.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:29.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:29.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:29.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:29.273 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:29.328 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:29.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:29.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:29.462 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:29.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:29.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:29.604 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:29.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:29.692 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:29.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:29.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:29.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:29.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:29.887 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:29.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:29.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:29 smithi179 ceph-mon[29608]: pgmap v162: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s 2024-03-24T11:23:29.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:29 smithi179 ceph-mon[37907]: pgmap v162: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s 2024-03-24T11:23:29.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:29 smithi181 ceph-mon[33998]: pgmap v162: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s 2024-03-24T11:23:29.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:30.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:30.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:30.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:30.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:30.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:30.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:30.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:23:30.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:30.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:30.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:30.455 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:30.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:30.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:30.597 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:30.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:30.640 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:30.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:30.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:30.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:30.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:30.884 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:30.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:30.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:31.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:31.117 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:31.177 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:31.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:31.310 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:31.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:31.441 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:31.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:31.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:31.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:31.703 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:31.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:31.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:23:31.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:31.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:31 smithi181 ceph-mon[33998]: pgmap v163: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:23:31.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:31 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:31.839 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:31 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:31.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:31 smithi179 ceph-mon[29608]: pgmap v163: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:23:31.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:31 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:31.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:31 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:31.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:31 smithi179 ceph-mon[37907]: pgmap v163: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:23:31.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:31 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:31.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:31 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.869 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:31.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:31.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:31.997 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:31.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:32.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:32.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:32.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:32.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:32.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:32.403 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:32.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:32.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:32.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.678 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:32.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:32.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:32.806 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:32.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:32.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:23:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:23:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:23:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:23:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:33.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:33.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:33.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:23:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:23:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:33.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:33.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:33.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.369 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:33.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:33.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:33.499 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:33.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:33.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:33.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:23:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:23:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:23:33.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:33.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:33.852 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:33 smithi179 ceph-mon[29608]: pgmap v164: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 964 op/s 2024-03-24T11:23:33.852 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1318869805' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:23:33.852 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:33 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:23:33.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:33 smithi179 ceph-mon[37907]: pgmap v164: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 964 op/s 2024-03-24T11:23:33.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1318869805' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:23:33.853 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:33 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:23:33.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:33.899 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:34.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:34.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:33 smithi181 ceph-mon[33998]: pgmap v164: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 964 op/s 2024-03-24T11:23:34.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1318869805' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:23:34.002 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:33 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:23:34.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:34.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:34.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:34.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:34.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:34.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:34.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:34.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.605 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:34.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:34.680 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:23:34.680 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:23:34.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.680 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:172881 7.7 7:eef07bbf:::benchmark_data_smithi179_78267_object53687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:34.666279+0000", 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.012725782, 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012758106, 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 172881 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:34.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666279+0000", 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666279+0000", 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:34.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666280+0000", 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3179999999999999e-06 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666285+0000", 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0799999999999996e-06 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666300+0000", 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4793e-05 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666308+0000", 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3340000000000002e-06 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666364+0000", 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5383000000000002e-05 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:34.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666497+0000", 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013325199999999999 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666626+0000", 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012865199999999999 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.672965+0000", 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063388810000000002 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.685 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:172884 7.7 7:ef842df6:::benchmark_data_smithi179_78267_object53690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:34.666447+0000", 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.012557398000000001, 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0126774, 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:34.686 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 172884 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666447+0000", 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666447+0000", 2024-03-24T11:23:34.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666448+0000", 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9899999999999999e-07 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666452+0000", 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.053e-06 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666470+0000", 2024-03-24T11:23:34.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7668000000000001e-05 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.666478+0000", 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.824e-06 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.669949+0000", 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003471853 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:34.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670074+0000", 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012494399999999999 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670533+0000", 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00045911500000000001 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675220+0000", 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046860970000000002 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:34.690 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:172887 7.7 7:ec901482:::benchmark_data_smithi179_78267_object53693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:34.670255+0000", 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0087498179999999995, 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089493249999999993, 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 172887 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:34.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670255+0000", 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670255+0000", 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670256+0000", 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.78e-07 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670260+0000", 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2830000000000003e-06 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670271+0000", 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0487e-05 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.670279+0000", 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0730000000000005e-06 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.672985+0000", 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002705926 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.673085+0000", 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100425 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:23:34.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.673194+0000", 2024-03-24T11:23:34.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108768 2024-03-24T11:23:34.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677945+0000", 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047514219999999999 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24698.0:172890 7.1 e125/92 7:8742479e:::benchmark_data_smithi179_78267_object53696:head v 125'10786, mlcod=125'10784)", 2024-03-24T11:23:34.696 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:34.674583+0000", 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0044215260000000003, 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046999310000000001, 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674583+0000", 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674583+0000", 2024-03-24T11:23:34.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674584+0000", 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7100000000000001e-07 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674589+0000", 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7160000000000002e-06 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674599+0000", 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0861e-05 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674605+0000", 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5799999999999999e-06 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674655+0000", 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0380000000000002e-05 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.674679+0000", 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3618e-05 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:172889 7.7 7:e985c706:::benchmark_data_smithi179_78267_object53695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:34.675131+0000", 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.003873675, 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042111800000000001, 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 172889 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675131+0000", 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675131+0000", 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675132+0000", 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9400000000000004e-07 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675136+0000", 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0010000000000003e-06 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675147+0000", 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1094000000000001e-05 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675153+0000", 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0090000000000004e-06 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675247+0000", 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4463999999999995e-05 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675354+0000", 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106968 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675461+0000", 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106789 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.678304+0000", 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028431020000000001 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.704 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:172894 7.d 7:b4177d65:::benchmark_data_smithi179_78267_object53700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:34.675876+0000", 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0031287609999999999, 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035435319999999998, 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:34.705 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 172894 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675876+0000", 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675876+0000", 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675877+0000", 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0410000000000001e-06 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675881+0000", 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4220000000000002e-06 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675897+0000", 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5933000000000001e-05 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.675905+0000", 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0739999999999993e-06 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.676074+0000", 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016856900000000001 2024-03-24T11:23:34.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.676180+0000", 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105976 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.676282+0000", 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000101809 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:34.709 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24698.0:172891 7.1 e125/92 7:819bd1ef:::benchmark_data_smithi179_78267_object53697:head v 125'10787, mlcod=125'10784)", 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:34.677337+0000", 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0016671660000000001, 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021567859999999999, 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:34.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677337+0000", 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677337+0000", 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677338+0000", 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0200000000000002e-07 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677343+0000", 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4280000000000004e-06 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677356+0000", 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2510999999999999e-05 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677363+0000", 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7150000000000001e-06 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677437+0000", 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3980000000000007e-05 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:34.677457+0000", 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0602e-05 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 7 2024-03-24T11:23:34.713 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:23:34.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:34.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:34.848 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:34.892 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:34.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:34 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:34 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:23:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:34 smithi181 ceph-mon[33998]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:23:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:34 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:23:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:34 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:23:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[29608]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:23:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[37907]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T11:23:34.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:23:34.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:34 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:23:34.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:35.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:35.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.162 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:35.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:35.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:35.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:35.323 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:23:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:23:35.380 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:35.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:35.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:35.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.583 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:35.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:35.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:23:35.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.732 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:35.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:35.833 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:35.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:35 smithi181 ceph-mon[33998]: pgmap v166: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-24T11:23:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:35 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:23:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:35 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:23:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:35 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:23:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:35 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:23:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:35 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:23:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:35 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:23:35.904 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:35.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[29608]: pgmap v166: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[37907]: pgmap v166: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-24T11:23:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:23:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:23:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:23:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:23:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:23:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:35 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:23:36.003 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:36.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:36.133 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:36.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.266 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:36.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:36.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.411 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:36.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:36.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:36.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:36.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:36.672 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:36.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:36.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:23:36.850 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:23:36.851 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:23:36.851 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:23:36.851 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:23:36.851 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:23:36.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[29608]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:23:36.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[29608]: 7.f scrub starts 2024-03-24T11:23:36.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:23:36.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:23:36.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:23:36.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:23:36.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[37907]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:23:36.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[37907]: 7.f scrub starts 2024-03-24T11:23:36.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:23:36.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:23:36.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:23:36.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:36 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:23:36.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:36.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:37.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:37.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:37.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:37.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:36 smithi181 ceph-mon[33998]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T11:23:37.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:36 smithi181 ceph-mon[33998]: 7.f scrub starts 2024-03-24T11:23:37.100 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:36 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:23:37.100 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:36 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:23:37.100 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:36 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:23:37.100 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:36 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:23:37.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:37.220 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:37.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:37.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:37.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:37.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:37.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:23:37.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:37.498 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:37.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:37.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:37.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:37.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:37.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:37.766 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:23:37.766 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:23:37.766 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:23:37.766 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:23:37.766 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:23:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:37.792 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:37.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:37.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:37.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:37.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[29608]: pgmap v168: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:23:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:23:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:23:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[29608]: 7.6 scrub starts 2024-03-24T11:23:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[37907]: pgmap v168: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:23:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:23:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:23:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:37 smithi179 ceph-mon[37907]: 7.6 scrub starts 2024-03-24T11:23:38.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:38.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:38.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:38.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:37 smithi181 ceph-mon[33998]: pgmap v168: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:23:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:37 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:23:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:37 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:23:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:37 smithi181 ceph-mon[33998]: 7.6 scrub starts 2024-03-24T11:23:38.155 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:38.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:38.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:23:38.235 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:23:38.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-24T11:23:38.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:38.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:38.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:38.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:38.460 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:38.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:38.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:38.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T11:23:38.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:38.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:38.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:38.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:38.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:38.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:38.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:38.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:38.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:38 smithi181 ceph-mon[33998]: 7.f scrub ok 2024-03-24T11:23:38.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:38 smithi181 ceph-mon[33998]: 7.1 scrub starts 2024-03-24T11:23:38.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:38 smithi179 ceph-mon[29608]: 7.f scrub ok 2024-03-24T11:23:38.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:38 smithi179 ceph-mon[29608]: 7.1 scrub starts 2024-03-24T11:23:38.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:38 smithi179 ceph-mon[37907]: 7.f scrub ok 2024-03-24T11:23:38.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:38 smithi179 ceph-mon[37907]: 7.1 scrub starts 2024-03-24T11:23:38.956 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:38.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.005 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:39.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:39.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.147 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:39.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:39.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:39.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:39.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:39.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.561 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:39.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:39.604 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:39.620 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:23:39.661 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:23:39.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:39.673 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:39.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:39.703 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:23:39.750 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:23:39.792 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:23:39.793 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:39.833 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:23:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.864 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:39.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:39.872 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: pgmap v169: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 713 op/s 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: 7.6 scrub ok 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:23:39.911 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: 4.17 scrub starts 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: 4.17 scrub ok 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: 7.1 scrub ok 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4036424573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: pgmap v169: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 713 op/s 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: 7.6 scrub ok 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:23:39.912 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:23:39.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: 4.17 scrub starts 2024-03-24T11:23:39.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: 4.17 scrub ok 2024-03-24T11:23:39.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: 7.1 scrub ok 2024-03-24T11:23:39.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:39 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4036424573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:39.913 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:23:39.917 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:23:39.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:39.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:39.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:40.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: pgmap v169: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 713 op/s 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: 7.6 scrub ok 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: 4.17 scrub starts 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: 4.17 scrub ok 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: 7.1 scrub ok 2024-03-24T11:23:40.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:39 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4036424573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:40.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:40.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:40.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:40.230 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:40.267 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:40.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:40.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:40.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:40.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:40.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:40.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:40.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:40.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:40.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:40.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:40.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:40.820 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:40.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:40.921 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:40.977 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:40.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:41.036 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:40 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:23:41.036 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:40 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:23:41.037 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:40 smithi181 ceph-mon[33998]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:23:41.037 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:40 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:23:41.037 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:40 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:23:41.037 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:40 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:23:41.037 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:40 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[29608]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:23:41.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:23:41.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[37907]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T11:23:41.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:23:41.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:23:41.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:23:41.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:40 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:23:41.079 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:41.101 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:23:41.101 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:23:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:41.125 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:41.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:41.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:41.276 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:41.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:41.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:41.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:41.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:41.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:41.560 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:41.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:41.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:41.703 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:41.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:41.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:41.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:41.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:41.914 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:41 smithi181 ceph-mon[33998]: pgmap v172: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 843 op/s 2024-03-24T11:23:41.914 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:41 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:23:41.914 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:41 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:23:41.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T11:23:41.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:41.989 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:41.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:42.091 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:42.132 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:42.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:41 smithi179 ceph-mon[29608]: pgmap v172: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 843 op/s 2024-03-24T11:23:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:41 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:23:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:41 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:23:42.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:41 smithi179 ceph-mon[37907]: pgmap v172: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 843 op/s 2024-03-24T11:23:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:41 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:23:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:41 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:23:42.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:42.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:42.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:42.277 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:42.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:42.415 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:42.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:42.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:42.554 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:42.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:42.657 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:42.700 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:42.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:42.801 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:42.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:42.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:42.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:42.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:23:42.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:42.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:43.022 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 7.c scrub starts 2024-03-24T11:23:43.022 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 6.b scrub starts 2024-03-24T11:23:43.022 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 6.b scrub ok 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 7.c scrub starts 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 6.b scrub starts 2024-03-24T11:23:43.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 6.b scrub ok 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:42 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:23:43.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:43.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 7.c scrub starts 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 6.b scrub starts 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 6.b scrub ok 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:23:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:42 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:23:43.178 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:43.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:43.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:43.330 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:43.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:43.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:43.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:43.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:43.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:23:43.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:43.617 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:43.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:43.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:43.778 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:43.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: pgmap v173: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 801 B/s rd, 0 B/s wr, 793 op/s 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: 7.c scrub ok 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:23:43.876 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:43 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:23:43.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:43.920 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:43.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:44.022 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:44.047 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:44.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:44.180 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:44.189 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:23:44.189 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: pgmap v173: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 801 B/s rd, 0 B/s wr, 793 op/s 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: 7.c scrub ok 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:23:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: pgmap v173: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 801 B/s rd, 0 B/s wr, 793 op/s 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: 7.c scrub ok 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:23:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:43 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:23:44.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.329 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:44.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:44.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.460 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:44.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:44.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:44.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:44.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:44.735 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:44.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:44.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:44.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[29608]: 7.3 scrub starts 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[29608]: pgmap v174: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:23:44.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:44.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[37907]: 7.3 scrub starts 2024-03-24T11:23:44.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:23:44.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:23:44.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[37907]: pgmap v174: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:23:44.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:23:44.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:44 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:23:44.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:44.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:44.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:45.094 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:45.139 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:45.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:45.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:45.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:44 smithi181 ceph-mon[33998]: 7.3 scrub starts 2024-03-24T11:23:45.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:44 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:23:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:44 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:23:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:44 smithi181 ceph-mon[33998]: pgmap v174: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:23:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:44 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:23:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:44 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:23:45.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:23:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:23:45.221 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:45.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:45.303 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:45.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:45.353 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:23:45.353 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:23:45.353 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:23:45.353 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:23:45.353 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:23:45.353 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:23:45.354 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:23:45.354 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:23:45.354 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:23:45.354 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:23:45.354 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:23:45.354 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:23:45.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:45.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:45.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:45.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:45.575 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:45.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:45.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:45.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:45.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:45.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:23:45.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:45.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:45.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T11:23:45.942 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:45.981 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:45.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:46.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:46.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:46.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:45 smithi181 ceph-mon[33998]: 3.1e scrub starts 2024-03-24T11:23:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:45 smithi181 ceph-mon[33998]: 3.1e scrub ok 2024-03-24T11:23:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:45 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:23:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:45 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:23:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:45 smithi181 ceph-mon[33998]: 7.3 scrub ok 2024-03-24T11:23:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:45 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/239870035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:45 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:46.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[29608]: 3.1e scrub starts 2024-03-24T11:23:46.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[29608]: 3.1e scrub ok 2024-03-24T11:23:46.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[29608]: 7.3 scrub ok 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/239870035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[37907]: 3.1e scrub starts 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[37907]: 3.1e scrub ok 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[37907]: 7.3 scrub ok 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/239870035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:46.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:45 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:23:46.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:46.266 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:46.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:46.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:23:46.360 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:23:46.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-24T11:23:46.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:46.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:46.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:46.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:46.602 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:46.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:46.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:46.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:46.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:46.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:46.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:46.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:46.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:47.056 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:47.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:47.129 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:23:47.129 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:23:47.129 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:47.129 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[29608]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:23:47.130 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[29608]: pgmap v176: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 695 B/s rd, 0 B/s wr, 652 op/s 2024-03-24T11:23:47.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:23:47.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:23:47.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:47.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[37907]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:23:47.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:46 smithi179 ceph-mon[37907]: pgmap v176: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 695 B/s rd, 0 B/s wr, 652 op/s 2024-03-24T11:23:47.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:46 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:23:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:46 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:23:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:46 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:23:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:46 smithi181 ceph-mon[33998]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T11:23:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:46 smithi181 ceph-mon[33998]: pgmap v176: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 695 B/s rd, 0 B/s wr, 652 op/s 2024-03-24T11:23:47.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:47.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:47.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:47.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:47.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:47.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:47.466 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:47.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:47.567 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:47.615 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:47.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:47.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:47.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:47.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:47.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:47.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:47.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:47.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:48.018 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:48.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:48.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:48.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:23:48.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:23:48.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:48.097 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:48.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:23:48.098 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:23:48.098 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:48.098 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:48.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:48.125 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:48.149 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:48.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:48.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:47 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:23:48.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:47 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:23:48.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:23:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:23:48.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:48.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:48.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:48.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:48.421 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:48.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:48.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:48.563 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:48.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:48.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:48.719 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:48.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:48.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:48.862 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:48.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:48.883 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:23:48.883 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:23:48.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.001 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:49.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:49.102 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:49.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.139 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:49.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:49.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:48 smithi181 ceph-mon[33998]: 7.7 scrub starts 2024-03-24T11:23:49.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:48 smithi181 ceph-mon[33998]: pgmap v177: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s 2024-03-24T11:23:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:48 smithi179 ceph-mon[29608]: 7.7 scrub starts 2024-03-24T11:23:49.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:48 smithi179 ceph-mon[29608]: pgmap v177: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s 2024-03-24T11:23:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:48 smithi179 ceph-mon[37907]: 7.7 scrub starts 2024-03-24T11:23:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:48 smithi179 ceph-mon[37907]: pgmap v177: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 689 op/s 2024-03-24T11:23:49.240 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:49.286 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:49.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:49.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:49.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:49.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:49.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:23:49.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:49.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:49.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T11:23:49.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:49.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:49.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:23:49.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:23:49.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 5.b deep-scrub starts 2024-03-24T11:23:49.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 5.b deep-scrub ok 2024-03-24T11:23:49.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 4.14 scrub starts 2024-03-24T11:23:49.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 4.14 scrub ok 2024-03-24T11:23:49.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:23:49.899 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:23:49.899 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:23:49.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:23:49.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:23:49.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 5.b deep-scrub starts 2024-03-24T11:23:49.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 5.b deep-scrub ok 2024-03-24T11:23:49.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 4.14 scrub starts 2024-03-24T11:23:49.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 4.14 scrub ok 2024-03-24T11:23:49.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:23:49.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:49 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:23:49.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:23:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:23:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 5.b deep-scrub starts 2024-03-24T11:23:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 5.b deep-scrub ok 2024-03-24T11:23:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 4.14 scrub starts 2024-03-24T11:23:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 4.14 scrub ok 2024-03-24T11:23:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:23:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:49 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:23:49.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:49.942 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:23:49.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:49.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:49.986 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:23:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:23:50.044 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:23:50.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:50.066 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:23:50.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:50.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:50.104 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:23:50.146 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:23:50.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:50.191 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:23:50.201 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:23:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:50.239 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:50.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:50.340 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:50.385 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:50.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:50.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:50.529 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:50.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:50.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:50.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:50.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:50.759 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:50.798 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:50.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:50.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:50.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:50.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:51.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:51.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:51.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:51.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:50 smithi181 ceph-mon[33998]: 7.7 scrub ok 2024-03-24T11:23:51.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4190068696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:51.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:50 smithi181 ceph-mon[33998]: pgmap v178: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:23:51.150 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:23:51.151 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:23:51.151 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:23:51.151 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:23:51.151 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:50 smithi179 ceph-mon[29608]: 7.7 scrub ok 2024-03-24T11:23:51.151 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4190068696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:51.151 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:50 smithi179 ceph-mon[29608]: pgmap v178: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s 2024-03-24T11:23:51.151 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:50 smithi179 ceph-mon[37907]: 7.7 scrub ok 2024-03-24T11:23:51.151 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4190068696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:23:51.152 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:50 smithi179 ceph-mon[37907]: pgmap v178: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s 2024-03-24T11:23:51.192 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:51.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:51.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:51.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:51.381 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:51.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:51.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:51.522 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:51.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:51.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:51.644 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:51.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:51.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:51.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:51.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:51.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:51.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:51.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:51.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:52.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:52.029 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:51 smithi181 ceph-mon[33998]: 2.1 deep-scrub starts 2024-03-24T11:23:52.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:51 smithi181 ceph-mon[33998]: 2.1 deep-scrub ok 2024-03-24T11:23:52.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:51 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:23:52.030 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:51 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:23:52.065 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:52.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:52.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:52.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:52.196 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[29608]: 2.1 deep-scrub starts 2024-03-24T11:23:52.197 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[29608]: 2.1 deep-scrub ok 2024-03-24T11:23:52.197 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:23:52.197 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:23:52.197 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[37907]: 2.1 deep-scrub starts 2024-03-24T11:23:52.197 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[37907]: 2.1 deep-scrub ok 2024-03-24T11:23:52.197 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:23:52.197 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:51 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:23:52.213 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:52.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:52.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:52.316 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:21748 7.5 7:a6556fc2:::benchmark_data_smithi179_78267_object21747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:15.731437+0000", 2024-03-24T11:23:52.326 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 156.59117003599999, 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068710555000000006, 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 21748 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731437+0000", 2024-03-24T11:23:52.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731437+0000", 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731438+0000", 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1990000000000001e-06 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731480+0000", 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1182e-05 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731490+0000", 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0695e-05 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731498+0000", 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0620000000000001e-06 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731554+0000", 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5297000000000003e-05 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731680+0000", 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126379 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731832+0000", 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151576 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.732818+0000", 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00098657699999999998 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.732939+0000", 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012116399999999999 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.800037+0000", 2024-03-24T11:23:52.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067097956 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.800094+0000", 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7043999999999998e-05 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.800148+0000", 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3424000000000002e-05 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.332 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26050 7.5 7:a02b24c6:::benchmark_data_smithi179_78267_object26049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 154.50321381800001, 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109015764, 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26050 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.333 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819394+0000", 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2999999999999999e-07 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819439+0000", 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4925000000000001e-05 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819450+0000", 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0586e-05 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819455+0000", 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7219999999999996e-06 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819502+0000", 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6295000000000003e-05 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819583+0000", 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0733000000000001e-05 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819680+0000", 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7677999999999996e-05 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820508+0000", 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000827445 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.821885+0000", 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001377365 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928313+0000", 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106428435 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928370+0000", 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6368e-05 2024-03-24T11:23:52.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928409+0000", 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9381999999999997e-05 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26055 7.5 7:a6f221c1:::benchmark_data_smithi179_78267_object26054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 154.50192223100001, 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10789344200000001, 2024-03-24T11:23:52.339 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26055 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820686+0000", 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2999999999999999e-07 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820747+0000", 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1148999999999993e-05 2024-03-24T11:23:52.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820760+0000", 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820768+0000", 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7610000000000007e-06 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820814+0000", 2024-03-24T11:23:52.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6644999999999998e-05 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820932+0000", 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117788 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.821060+0000", 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012757900000000001 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.822115+0000", 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010555689999999999 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.822131+0000", 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5673000000000001e-05 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928469+0000", 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10633761 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928531+0000", 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2549e-05 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928578+0000", 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7308000000000002e-05 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.345 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26494 7.5 7:a4c9ac9c:::benchmark_data_smithi179_78267_object26493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:52.346 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:23:52.346 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 154.19934256499999, 2024-03-24T11:23:52.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15363169800000001, 2024-03-24T11:23:52.346 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.346 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.346 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.346 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26494 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123266+0000", 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1710000000000001e-06 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123350+0000", 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3776000000000006e-05 2024-03-24T11:23:52.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123366+0000", 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6779999999999999e-05 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123375+0000", 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.439e-06 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123428+0000", 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2760000000000003e-05 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123535+0000", 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107574 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123703+0000", 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016783 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.124976+0000", 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012726930000000001 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125304+0000", 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032807300000000001 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276785+0000", 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.151481123 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276851+0000", 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6204999999999994e-05 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276896+0000", 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5274000000000001e-05 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.352 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26500 7.5 7:afbe2af8:::benchmark_data_smithi179_78267_object26499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 154.19683095299999, 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15542166600000001, 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26500 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125777+0000", 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7100000000000011e-07 2024-03-24T11:23:52.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125824+0000", 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6388999999999999e-05 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125835+0000", 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1032999999999999e-05 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125840+0000", 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6099999999999997e-06 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125886+0000", 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5711000000000003e-05 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125998+0000", 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111952 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126130+0000", 2024-03-24T11:23:52.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013231699999999999 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127017+0000", 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00088662500000000004 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127330+0000", 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000313006 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281101+0000", 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15377119 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281159+0000", 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7759e-05 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281198+0000", 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9103000000000002e-05 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.359 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:27107 7.5 7:a4201e14:::benchmark_data_smithi179_78267_object27106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.543620+0000", 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 153.778987478, 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067854379000000006, 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 27107 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.360 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543620+0000", 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543620+0000", 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543621+0000", 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8100000000000001e-07 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543688+0000", 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7092000000000001e-05 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543701+0000", 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2741e-05 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543709+0000", 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543778+0000", 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9062999999999999e-05 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543902+0000", 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012383000000000001 2024-03-24T11:23:52.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.544060+0000", 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015877299999999999 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.545099+0000", 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001038773 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.547163+0000", 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020637400000000001 2024-03-24T11:23:52.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.611359+0000", 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.064196290000000003 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.611426+0000", 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7287999999999997e-05 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.611474+0000", 2024-03-24T11:23:52.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.774e-05 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:31022 7.5 7:a6087dad:::benchmark_data_smithi179_78267_object31021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:20.812672+0000", 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 151.50993473400001, 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078745411000000001, 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.366 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 31022 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812672+0000", 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812672+0000", 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812674+0000", 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.037e-06 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812719+0000", 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5053999999999999e-05 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812744+0000", 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5018999999999999e-05 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812752+0000", 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4449999999999996e-06 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812818+0000", 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5874e-05 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812927+0000", 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010874600000000001 2024-03-24T11:23:52.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.813106+0000", 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000179459 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.814261+0000", 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011543949999999999 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.818357+0000", 2024-03-24T11:23:52.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004096939 2024-03-24T11:23:52.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.891317+0000", 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072959780000000002 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.891369+0000", 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1279000000000002e-05 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.891418+0000", 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9384e-05 2024-03-24T11:23:52.372 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:64062 7.5 7:a7020b65:::benchmark_data_smithi179_78267_object64061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:42.573650+0000", 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 129.748957316, 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078293327999999995, 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.373 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 64062 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573650+0000", 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573650+0000", 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573651+0000", 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.251e-06 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573696+0000", 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5191999999999997e-05 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573709+0000", 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.242e-05 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573718+0000", 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9360000000000005e-06 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573786+0000", 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8387999999999996e-05 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573886+0000", 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9471999999999999e-05 2024-03-24T11:23:52.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573989+0000", 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010310300000000001 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.576159+0000", 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021701720000000002 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.577205+0000", 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010460350000000001 2024-03-24T11:23:52.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.651809+0000", 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074603931999999998 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.651881+0000", 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1705000000000005e-05 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.651943+0000", 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2722e-05 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:66218 7.5 7:a1d02a05:::benchmark_data_smithi179_78267_object66217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 127.69821490699999, 2024-03-24T11:23:52.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087797980999999997, 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 66218 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624393+0000", 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0869999999999999e-06 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624444+0000", 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0929e-05 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624454+0000", 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4730000000000006e-06 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624460+0000", 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7180000000000001e-06 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624515+0000", 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5519999999999997e-05 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624671+0000", 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015631299999999999 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624776+0000", 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104378 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.628751+0000", 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039755060000000002 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.686471+0000", 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057719422999999999 2024-03-24T11:23:52.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712076+0000", 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025605297999999999 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712140+0000", 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4049999999999998e-05 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712190+0000", 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0285999999999999e-05 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:66219 7.5 7:a6ed2970:::benchmark_data_smithi179_78267_object66218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 127.69812226499999, 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087858568999999997, 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 66219 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:23:52.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624486+0000", 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8199999999999999e-07 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624563+0000", 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6958000000000004e-05 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624571+0000", 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3319999999999992e-06 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624577+0000", 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1619999999999996e-06 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626484+0000", 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019068620000000001 2024-03-24T11:23:52.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626623+0000", 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013903099999999999 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626786+0000", 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000163096 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.628876+0000", 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020895620000000001 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.686625+0000", 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057748762000000002 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712252+0000", 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025627644000000002 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712307+0000", 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4877000000000003e-05 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712344+0000", 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6501000000000002e-05 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:68243 7.5 7:a17adc7b:::benchmark_data_smithi179_78267_object68242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:46.516297+0000", 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.80630992899999, 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067671549999999997, 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 68243 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516297+0000", 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516297+0000", 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516299+0000", 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6959999999999999e-06 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516353+0000", 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4018999999999999e-05 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516368+0000", 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5175e-05 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516377+0000", 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3450000000000006e-06 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516444+0000", 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7347000000000004e-05 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516614+0000", 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017001700000000001 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516801+0000", 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018701300000000001 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521912+0000", 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051114369999999999 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521980+0000", 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7380999999999997e-05 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.583889+0000", 2024-03-24T11:23:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.061909750999999999 2024-03-24T11:23:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.583938+0000", 2024-03-24T11:23:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8121000000000002e-05 2024-03-24T11:23:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.583969+0000", 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1248000000000003e-05 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:68248 7.5 7:a886aae0:::benchmark_data_smithi179_78267_object68247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:46.517950+0000", 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.804656924, 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066190224000000006, 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.401 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 68248 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517950+0000", 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517950+0000", 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517952+0000", 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6500000000000001e-06 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517982+0000", 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9611000000000002e-05 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517997+0000", 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5104e-05 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.518005+0000", 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5339999999999999e-06 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.519133+0000", 2024-03-24T11:23:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001127798 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.519293+0000", 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015965599999999999 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.519505+0000", 2024-03-24T11:23:52.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000212015 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521971+0000", 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024665149999999999 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521989+0000", 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7584e-05 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.584016+0000", 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062027269000000003 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.584097+0000", 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1283000000000001e-05 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.584141+0000", 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3204999999999997e-05 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.407 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:68251 7.5 7:ab028aee:::benchmark_data_smithi179_78267_object68250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:46.521793+0000", 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 125.80081471699999, 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.070321116000000003, 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.408 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 68251 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521793+0000", 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521793+0000", 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521794+0000", 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.189e-06 2024-03-24T11:23:52.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521865+0000", 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1311999999999998e-05 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521881+0000", 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6220000000000001e-05 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521898+0000", 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.683e-05 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.522012+0000", 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011431500000000001 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.522162+0000", 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014995399999999999 2024-03-24T11:23:52.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.522334+0000", 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171767 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.524633+0000", 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022989410000000001 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.524759+0000", 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126107 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.591882+0000", 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067122845 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.591939+0000", 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7432000000000002e-05 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.592114+0000", 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017420399999999999 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:70839 7.5 7:a954ad76:::benchmark_data_smithi179_78267_object70838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:48.814138+0000", 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 123.508469252, 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074097784999999999, 2024-03-24T11:23:52.415 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 70839 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814138+0000", 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814138+0000", 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814140+0000", 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0679999999999999e-06 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814189+0000", 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8749999999999999e-05 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814204+0000", 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5238e-05 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814214+0000", 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0028e-05 2024-03-24T11:23:52.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814290+0000", 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6008999999999997e-05 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814451+0000", 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160877 2024-03-24T11:23:52.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814607+0000", 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015562200000000001 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.817243+0000", 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026363760000000002 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.817385+0000", 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014165299999999999 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.888075+0000", 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.070690826999999998 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.888176+0000", 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000101055 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.888236+0000", 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9281999999999999e-05 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:75763 7.5 7:a2642e9d:::benchmark_data_smithi179_78267_object75762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:53.173290+0000", 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 119.14931712400001, 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072918858000000003, 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.422 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 75763 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173290+0000", 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173290+0000", 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173292+0000", 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.395e-06 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173346+0000", 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4299000000000003e-05 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173360+0000", 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4133e-05 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173366+0000", 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9839999999999998e-06 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173441+0000", 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5160000000000005e-05 2024-03-24T11:23:52.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173556+0000", 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114734 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173659+0000", 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010362 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.176081+0000", 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024217409999999998 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.177078+0000", 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00099639200000000002 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.246093+0000", 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069015100999999995 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.246157+0000", 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4659999999999994e-05 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.246209+0000", 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1638999999999998e-05 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.430 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:92829 7.5 7:a6a4baa4:::benchmark_data_smithi179_78267_object92828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:09.778359+0000", 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 102.544248305, 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.077760105999999996, 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 92829 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778359+0000", 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778359+0000", 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778361+0000", 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.669e-06 2024-03-24T11:23:52.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778418+0000", 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7343999999999998e-05 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778435+0000", 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7329999999999998e-05 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778444+0000", 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7339999999999997e-06 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778510+0000", 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6383999999999993e-05 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778661+0000", 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000150894 2024-03-24T11:23:52.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778822+0000", 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160421 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.782048+0000", 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003226745 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.783212+0000", 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011637469999999999 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.856010+0000", 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072798013999999994 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.856075+0000", 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4653999999999995e-05 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.856119+0000", 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4169999999999999e-05 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:92981 7.5 7:aa6b41c2:::benchmark_data_smithi179_78267_object92980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:09.961306+0000", 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 102.36130160800001, 2024-03-24T11:23:52.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082104115000000005, 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 92981 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961306+0000", 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961306+0000", 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961307+0000", 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9399999999999993e-07 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961420+0000", 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000113791 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961457+0000", 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7027999999999999e-05 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961469+0000", 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1399999999999999e-05 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.962674+0000", 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001205649 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.962829+0000", 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000154368 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.962986+0000", 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015717899999999999 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.965368+0000", 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023822230000000002 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.966487+0000", 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011184700000000001 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:10.043332+0000", 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.076844777000000003 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:10.043374+0000", 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2055999999999998e-05 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:10.043410+0000", 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6180000000000003e-05 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.443 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:154476 7.5 7:a41c9bf3:::benchmark_data_smithi179_78267_object35282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:15.098300+0000", 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.224307670999998, 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.065449867999999994, 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 154476 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098300+0000", 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098300+0000", 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098301+0000", 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.316e-06 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098304+0000", 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4999999999999999e-06 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098316+0000", 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2038000000000001e-05 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098323+0000", 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9190000000000002e-06 2024-03-24T11:23:52.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098380+0000", 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7105000000000003e-05 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098648+0000", 2024-03-24T11:23:52.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026745099999999998 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098772+0000", 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012398699999999999 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.102165+0000", 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033926730000000001 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.107518+0000", 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053531539999999997 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.163637+0000", 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056119166999999998 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.163704+0000", 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6909000000000006e-05 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.163749+0000", 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5649000000000003e-05 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:177111 7.5 7:ad9fdd86:::benchmark_data_smithi179_78267_object57917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-24T11:23:52.450 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:38.969858+0000", 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.352749727000001, 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066590238999999996, 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177111 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969858+0000", 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969858+0000", 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969859+0000", 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5060000000000001e-06 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969865+0000", 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7010000000000001e-06 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969880+0000", 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.573e-05 2024-03-24T11:23:52.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969890+0000", 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8179999999999999e-06 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969971+0000", 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0487000000000004e-05 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.970125+0000", 2024-03-24T11:23:52.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015467899999999999 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.970278+0000", 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015304699999999999 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.973268+0000", 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029894129999999998 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.992670+0000", 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019401728 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:39.036295+0000", 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043625821000000002 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:39.036399+0000", 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103675 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:39.036448+0000", 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8634000000000002e-05 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:188659 7.5 7:a500a61f:::benchmark_data_smithi179_78267_object69465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:50.651718+0000", 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.670889158, 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.064966386000000001, 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:23:52.458 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 188659 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.651718+0000", 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.651718+0000", 2024-03-24T11:23:52.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.651719+0000", 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9700000000000005e-07 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.651731+0000", 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2211999999999999e-05 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.678656+0000", 2024-03-24T11:23:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.026924601999999999 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.678671+0000", 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5356999999999999e-05 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.684422+0000", 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057505630000000002 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:23:52.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.686972+0000", 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025506679999999999 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.687103+0000", 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000130214 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.692818+0000", 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057151789999999999 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.692939+0000", 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120951 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.716620+0000", 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023680898999999998 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.716654+0000", 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4214999999999997e-05 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:23:52.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:50.716684+0000", 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0629000000000001e-05 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:23:52.464 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:52.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:52.566 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:52.607 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:52.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:52.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:52.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:52.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:52.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:52.886 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:52.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:23:52.969 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:23:52.988 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.029 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:53.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:53.129 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:23:53.129 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[29608]: 4.13 deep-scrub starts 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[29608]: 4.13 deep-scrub ok 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[29608]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[37907]: 4.13 deep-scrub starts 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[37907]: 4.13 deep-scrub ok 2024-03-24T11:23:53.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:52 smithi179 ceph-mon[37907]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s 2024-03-24T11:23:53.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:53.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:52 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:23:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:52 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:23:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:52 smithi181 ceph-mon[33998]: 4.13 deep-scrub starts 2024-03-24T11:23:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:52 smithi181 ceph-mon[33998]: 4.13 deep-scrub ok 2024-03-24T11:23:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:52 smithi181 ceph-mon[33998]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s 2024-03-24T11:23:53.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:53.261 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:53.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:53.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:23:53.391 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.432 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:53.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:53.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:53.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:53.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:23:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:23:53.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:53.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:53.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:23:53.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:53.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:53.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:53.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:53.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:54.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:54.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:54.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:54.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:53 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:23:54.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:53 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:23:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:53 smithi181 ceph-mon[33998]: 7.8 scrub starts 2024-03-24T11:23:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:53 smithi181 ceph-mon[33998]: 5.1f deep-scrub starts 2024-03-24T11:23:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:53 smithi181 ceph-mon[33998]: 5.1f deep-scrub ok 2024-03-24T11:23:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:53 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:54.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:23:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:23:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[37907]: 7.8 scrub starts 2024-03-24T11:23:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[37907]: 5.1f deep-scrub starts 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[37907]: 5.1f deep-scrub ok 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[29608]: 7.8 scrub starts 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[29608]: 5.1f deep-scrub starts 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[29608]: 5.1f deep-scrub ok 2024-03-24T11:23:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:53 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:23:54.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:54.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:54.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:54.366 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:54.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:54.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:54.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:54.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:54.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:54.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:54.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:54.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:54.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:54.784 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:54.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:54.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:54.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:54.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:55.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:55.059 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:55.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:54 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:23:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:54 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:23:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:54 smithi181 ceph-mon[33998]: 7.8 scrub ok 2024-03-24T11:23:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:54 smithi181 ceph-mon[33998]: pgmap v180: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T11:23:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:54 smithi181 ceph-mon[33998]: 4.8 deep-scrub starts 2024-03-24T11:23:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:54 smithi181 ceph-mon[33998]: 4.8 deep-scrub ok 2024-03-24T11:23:55.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:55.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:55.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:55.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:23:55.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:23:55.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[29608]: 7.8 scrub ok 2024-03-24T11:23:55.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[29608]: pgmap v180: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[29608]: 4.8 deep-scrub starts 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[29608]: 4.8 deep-scrub ok 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[37907]: 7.8 scrub ok 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[37907]: pgmap v180: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[37907]: 4.8 deep-scrub starts 2024-03-24T11:23:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:54 smithi179 ceph-mon[37907]: 4.8 deep-scrub ok 2024-03-24T11:23:55.211 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:23:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:23:55.292 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:55.337 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:55.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:55.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:55.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:55.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:55.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:55.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:55.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:55.699 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:55.746 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:55.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:55.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:55.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:55.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:55.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:56.011 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:56.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:56.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:55 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:23:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:55 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:23:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:55 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:23:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:55 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:23:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.153 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:56.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:56.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:23:56.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:23:56.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:23:56.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:23:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:23:56.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:23:56.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:23:56.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:55 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:23:56.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:56.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:56.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.428 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:56.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:56.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:56.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:56.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:56.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:56.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:56.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:56.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:56.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:56.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:56.950 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:56.966 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:23:57.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:23:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:57.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:57.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:23:57.079 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:56 smithi179 ceph-mon[29608]: 7.d scrub starts 2024-03-24T11:23:57.079 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:56 smithi179 ceph-mon[29608]: pgmap v181: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 987 op/s 2024-03-24T11:23:57.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:56 smithi179 ceph-mon[37907]: 7.d scrub starts 2024-03-24T11:23:57.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:56 smithi179 ceph-mon[37907]: pgmap v181: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 987 op/s 2024-03-24T11:23:57.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:56 smithi181 ceph-mon[33998]: 7.d scrub starts 2024-03-24T11:23:57.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:56 smithi181 ceph-mon[33998]: pgmap v181: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 987 op/s 2024-03-24T11:23:57.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:57.232 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:57.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:57.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:57.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:57.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:57.461 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:57.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:57.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:57.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:57.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:57.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:57.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:57.781 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:57.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:57.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:57.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:57.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:23:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:23:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[29608]: 7.d scrub ok 2024-03-24T11:23:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[29608]: 7.e scrub starts 2024-03-24T11:23:57.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:23:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:23:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[37907]: 7.d scrub ok 2024-03-24T11:23:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:57 smithi179 ceph-mon[37907]: 7.e scrub starts 2024-03-24T11:23:57.970 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:23:57.970 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:23:57.971 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-24T11:23:57.971 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:23:58.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:58.056 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:58.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:58.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:58.183 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:58.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:58.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:58.343 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:58.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:58.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:57 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:23:58.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:57 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:23:58.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:57 smithi181 ceph-mon[33998]: 7.d scrub ok 2024-03-24T11:23:58.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:57 smithi181 ceph-mon[33998]: 7.e scrub starts 2024-03-24T11:23:58.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:23:58.471 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:58.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:23:58.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:58.615 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:58.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:58.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:58.759 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:58.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:58.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:58.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:23:58.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:58.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:23:58.933 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:23:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 926 op/s 2024-03-24T11:23:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: 7.e scrub ok 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3533861851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: 6.1d deep-scrub starts 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[29608]: 6.1d deep-scrub ok 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 926 op/s 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: 7.e scrub ok 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:23:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3533861851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:58.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:58.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: 6.1d deep-scrub starts 2024-03-24T11:23:58.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:58 smithi179 ceph-mon[37907]: 6.1d deep-scrub ok 2024-03-24T11:23:58.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:59.029 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:59.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:59.130 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:23:59.170 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:59.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:23:59.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:23:59.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:59.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:23:59.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 926 op/s 2024-03-24T11:23:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: 7.e scrub ok 2024-03-24T11:23:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:23:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:23:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3533861851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:23:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: 6.1d deep-scrub starts 2024-03-24T11:23:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:58 smithi181 ceph-mon[33998]: 6.1d deep-scrub ok 2024-03-24T11:23:59.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:23:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:59.454 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:59.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:23:59.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:59.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:59.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:59.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:59.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:23:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:59.766 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:23:59.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:23:59.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:23:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:59.893 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:23:59.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:23:59.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:00.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:00.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:00.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:00.201 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:00.201 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:24:00.202 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[29608]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:24:00.202 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:24:00.202 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:00.202 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:24:00.202 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[37907]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:24:00.202 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:23:59 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:24:00.259 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:00.308 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:00.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:00.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:59 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:00.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:59 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:24:00.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:59 smithi181 ceph-mon[33998]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T11:24:00.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:23:59 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:24:00.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:00.440 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:00.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.586 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:00.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:00.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:00.713 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:00.713 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:00.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:00.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:00.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:00.982 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:00.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:01.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:01.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:01.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:01.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:00 smithi179 ceph-mon[29608]: pgmap v184: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 980 op/s 2024-03-24T11:24:01.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:00 smithi179 ceph-mon[37907]: pgmap v184: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 980 op/s 2024-03-24T11:24:01.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:01.263 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:01.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:01.303 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:00 smithi181 ceph-mon[33998]: pgmap v184: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 980 op/s 2024-03-24T11:24:01.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:01.405 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:01.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:01.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:01.541 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:01.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:01.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:01.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:01.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:01.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:24:01.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:01.841 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:01.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:01.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:01.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:01.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:01.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:02.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:02.126 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:02.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:02.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:02.267 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:02.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:02.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:02.410 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:02.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:02.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:02.576 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:02.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200279 7.5 e130/92 7:a71c56be:::benchmark_data_smithi179_78267_object81085:head v 130'12604, mlcod=130'12602)", 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.658474+0000", 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0054657839999999996, 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054963649999999996, 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.666 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658474+0000", 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658474+0000", 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658475+0000", 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4199999999999995e-07 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658483+0000", 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4339999999999992e-06 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658493+0000", 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4620000000000002e-06 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658499+0000", 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0329999999999997e-06 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658555+0000", 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6357000000000003e-05 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.658573+0000", 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7467999999999999e-05 2024-03-24T11:24:02.669 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200280 7.5 e130/92 7:ab1b4604:::benchmark_data_smithi179_78267_object81086:head v 130'12605, mlcod=130'12602)", 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.660710+0000", 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0032296180000000001, 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033303769999999998, 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.660710+0000", 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.660710+0000", 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.660711+0000", 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6599999999999996e-07 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.660721+0000", 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4280000000000002e-06 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.660729+0000", 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8200000000000003e-06 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.660737+0000", 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4560000000000003e-06 2024-03-24T11:24:02.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.662020+0000", 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00128338 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.662040+0000", 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0051000000000002e-05 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200278 7.e e130/92 7:7fe1f2d0:::benchmark_data_smithi179_78267_object81084:head v 130'12675, mlcod=130'12672)", 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.661336+0000", 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0026040310000000001, 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027659680000000002, 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661336+0000", 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661336+0000", 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661337+0000", 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5400000000000001e-07 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661342+0000", 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9969999999999998e-06 2024-03-24T11:24:02.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661353+0000", 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0937e-05 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661359+0000", 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7479999999999996e-06 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:02.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.662069+0000", 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000709744 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.662087+0000", 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8405000000000001e-05 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200281 7.d e130/92)", 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.661385+0000", 2024-03-24T11:24:02.677 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0025550659999999999, 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027768760000000002, 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661385+0000", 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661385+0000", 2024-03-24T11:24:02.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661385+0000", 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4900000000000001e-07 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661389+0000", 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6049999999999998e-06 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661398+0000", 2024-03-24T11:24:02.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.465e-06 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661403+0000", 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7940000000000002e-06 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.680 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200282 7.7 e130/92)", 2024-03-24T11:24:02.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:02.681 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.661613+0000", 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0023268120000000002, 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025941010000000001, 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661613+0000", 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661613+0000", 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661614+0000", 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0200000000000003e-07 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661620+0000", 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9789999999999998e-06 2024-03-24T11:24:02.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661627+0000", 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3919999999999997e-06 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.661632+0000", 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4269999999999999e-06 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200283 7.5 e130/92)", 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.663458+0000", 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00048215600000000002, 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000796651, 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.663458+0000", 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.663458+0000", 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.663459+0000", 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8700000000000004e-07 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.663468+0000", 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4329999999999993e-06 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.663479+0000", 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1109e-05 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.663486+0000", 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0709999999999997e-06 2024-03-24T11:24:02.687 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:200269 7.6 7:623103b4:::benchmark_data_smithi179_78267_object81075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.648229+0000", 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.015710775, 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016078727000000001, 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:02.688 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 200269 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648229+0000", 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648229+0000", 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648230+0000", 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1999999999999998e-07 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648234+0000", 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.546e-06 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648247+0000", 2024-03-24T11:24:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3027000000000001e-05 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648255+0000", 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5220000000000007e-06 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648294+0000", 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8417999999999997e-05 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648363+0000", 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9746000000000006e-05 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.648430+0000", 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6557999999999995e-05 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.653936+0000", 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0055060680000000002 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657519+0000", 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035833689999999999 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:200273 7.6 7:6a6d69bf:::benchmark_data_smithi179_78267_object81079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.652700+0000", 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.011239846, 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011694816, 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:24:02.693 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 200273 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.652700+0000", 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.652700+0000", 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.652702+0000", 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4529999999999999e-06 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.652706+0000", 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4689999999999999e-06 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.652718+0000", 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2091e-05 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.652726+0000", 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9549999999999992e-06 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.653965+0000", 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001238844 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.654078+0000", 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000113068 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.654190+0000", 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000111871 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657562+0000", 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033720769999999998 2024-03-24T11:24:02.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657682+0000", 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012000799999999999 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200277 7.d e130/92 7:b5a336e9:::benchmark_data_smithi179_78267_object81083:head v 130'12471, mlcod=130'12469)", 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.655918+0000", 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00802164, 2024-03-24T11:24:02.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0085692939999999999, 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.655918+0000", 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.655918+0000", 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.655919+0000", 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9999999999999998e-07 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.655925+0000", 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6499999999999999e-06 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.655933+0000", 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2280000000000001e-06 2024-03-24T11:24:02.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.655938+0000", 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8040000000000001e-06 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.656583+0000", 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00064580400000000004 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.656603+0000", 2024-03-24T11:24:02.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9936999999999999e-05 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:200275 7.e e130/92 7:741c1925:::benchmark_data_smithi179_78267_object81081:head v 130'12674, mlcod=130'12672)", 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:02.657510+0000", 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0064296869999999999, 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070449459999999998, 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:02.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657510+0000", 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657510+0000", 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657511+0000", 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7199999999999998e-07 2024-03-24T11:24:02.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657517+0000", 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7420000000000003e-06 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657528+0000", 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0828e-05 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657535+0000", 2024-03-24T11:24:02.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9809999999999997e-06 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657594+0000", 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9555999999999997e-05 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:02.657612+0000", 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7638e-05 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:02.705 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:24:02.706 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 10 2024-03-24T11:24:02.706 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:24:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:02.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:02.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:02.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:02.884 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:02.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:02.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.016 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:03.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:03.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:03.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:03.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.296 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:03.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:03.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:03.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:24:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:24:03.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:03.525 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:03.556 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:03 smithi179 ceph-mon[29608]: pgmap v185: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 980 op/s 2024-03-24T11:24:03.556 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:03 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:03.557 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:03.557 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:03 smithi179 ceph-mon[37907]: pgmap v185: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 980 op/s 2024-03-24T11:24:03.557 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:03 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:03.557 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.571 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:03.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:03.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.721 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:03.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:03.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:03.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:03 smithi181 ceph-mon[33998]: pgmap v185: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 980 op/s 2024-03-24T11:24:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:03 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:03.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:03.996 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:03.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:04.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:04.134 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:04.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:04.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:04.263 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:04.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:04.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:04.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:04.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:04.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:24:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:24:04.502 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:24:04.503 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:24:04.503 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:24:04.503 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:24:04.503 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:24:04.503 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:24:04.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:04.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:04.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:04.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:04.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:04.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4074582183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:04.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:04.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4074582183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:04.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:04.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:04.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:04.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4074582183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:04.921 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:04.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:05.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:05.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:05.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:05.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:05.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:05.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:05.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:05.322 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:05.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:05.378 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:24:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:24:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:24:05.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:05.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:05.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:05.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:05.583 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:05.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:05.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:05 smithi179 ceph-mon[29608]: pgmap v186: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 955 op/s 2024-03-24T11:24:05.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:05 smithi179 ceph-mon[37907]: pgmap v186: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 955 op/s 2024-03-24T11:24:05.714 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:05.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:05.816 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:05.859 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:05.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:05.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:24:05.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:05 smithi181 ceph-mon[33998]: pgmap v186: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 955 op/s 2024-03-24T11:24:05.950 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:05.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.014 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:06.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:06.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:06.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:06.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:06.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:06.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:06.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:06.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:06.546 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:06.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:06.674 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:06.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.823 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:06.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:06.924 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:06.968 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:06.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:07.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:07.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:07.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:07.140 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:24:07.145 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:24:07.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:07.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:07.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:07.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:07.361 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:07.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:07.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:24:07.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:07.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:07.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:07.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:07.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:07.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:07.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:07.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:07.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:07.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:07 smithi181 ceph-mon[33998]: pgmap v187: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 955 op/s 2024-03-24T11:24:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:07 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:24:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:07 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:24:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:07 smithi181 ceph-mon[33998]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:24:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:07.940 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:07.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[29608]: pgmap v187: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 955 op/s 2024-03-24T11:24:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:24:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:24:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[29608]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:24:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[37907]: pgmap v187: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 955 op/s 2024-03-24T11:24:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:24:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:24:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:07 smithi179 ceph-mon[37907]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T11:24:08.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:08.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:08.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:08.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:08.202 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:08.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:08.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:08.334 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:08.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:08.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:08.480 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:08.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:08.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:08.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:08.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:08.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:08.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:08.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:08.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:08.878 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:08.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:08.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:08 smithi181 ceph-mon[33998]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:24:08.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:08 smithi179 ceph-mon[37907]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:24:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:08 smithi179 ceph-mon[29608]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:24:08.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:09.008 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:09.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:09.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:09.150 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:09.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:09.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:09.296 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:09.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:09.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:09.434 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:09.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:09.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:09.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:09.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:09.618 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-03-24T11:24:06.496392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2453018998,"num_objects":37620,"num_object_clones":0,"num_object_copies":112860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37620,"num_whiteouts":0,"num_read":1237,"num_read_kb":1083,"num_write":201577,"num_write_kb":7630686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90763,"ondisk_log_size":90763,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8194632,"kb_used_data":7313416,"kb_used_omap":12,"kb_used_meta":881011,"kb_avail":741766584,"statfs":{"total":767960285184,"available":759568982016,"internally_reserved":0,"allocated":7488937984,"data_stored":7486359530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":902155825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,2,7,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-564658176,"num_objects":-8616,"num_object_clones":0,"num_object_copies":-25848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8616,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8616,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.399479"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.912648+0000","last_change":"2024-03-24T11:22:04.325892+0000","last_active":"2024-03-24T11:23:58.912648+0000","last_peered":"2024-03-24T11:23:58.912648+0000","last_clean":"2024-03-24T11:23:58.912648+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:23:58.912648+0000","last_undegraded":"2024-03-24T11:23:58.912648+0000","last_fullsized":"2024-03-24T11:23:58.912648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:04.325807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:04.325807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:07.078164+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.924139+0000","last_change":"2024-03-24T11:23:52.760709+0000","last_active":"2024-03-24T11:23:58.924139+0000","last_peered":"2024-03-24T11:23:58.924139+0000","last_clean":"2024-03-24T11:23:58.924139+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:23:58.924139+0000","last_undegraded":"2024-03-24T11:23:58.924139+0000","last_fullsized":"2024-03-24T11:23:58.924139+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:52.760278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:52.760278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:10.599945+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.950044+0000","last_change":"2024-03-24T11:22:51.453107+0000","last_active":"2024-03-24T11:23:58.950044+0000","last_peered":"2024-03-24T11:23:58.950044+0000","last_clean":"2024-03-24T11:23:58.950044+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:23:58.950044+0000","last_undegraded":"2024-03-24T11:23:58.950044+0000","last_fullsized":"2024-03-24T11:23:58.950044+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:51.452998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:51.452998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:07.212472+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":185,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.913057+0000","last_change":"2024-03-24T11:22:11.342339+0000","last_active":"2024-03-24T11:23:58.913057+0000","last_peered":"2024-03-24T11:23:58.913057+0000","last_clean":"2024-03-24T11:23:58.913057+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:23:58.913057+0000","last_undegraded":"2024-03-24T11:23:58.913057+0000","last_fullsized":"2024-03-24T11:23:58.913057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:22:11.342021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:11.342021+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":137,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919747+0000","last_change":"2024-03-24T11:21:49.304312+0000","last_active":"2024-03-24T11:23:58.919747+0000","last_peered":"2024-03-24T11:23:58.919747+0000","last_clean":"2024-03-24T11:23:58.919747+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:23:58.919747+0000","last_undegraded":"2024-03-24T11:23:58.919747+0000","last_fullsized":"2024-03-24T11:23:58.919747+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:49.304197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:21:49.304197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.913194+0000","last_change":"2024-03-24T11:22:57.211619+0000","last_active":"2024-03-24T11:23:58.913194+0000","last_peered":"2024-03-24T11:23:58.913194+0000","last_clean":"2024-03-24T11:23:58.913194+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:23:58.913194+0000","last_undegraded":"2024-03-24T11:23:58.913194+0000","last_fullsized":"2024-03-24T11:23:58.913194+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:57.211503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:57.211503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:25.790116+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919815+0000","last_change":"2024-03-24T11:23:44.804746+0000","last_active":"2024-03-24T11:23:58.919815+0000","last_peered":"2024-03-24T11:23:58.919815+0000","last_clean":"2024-03-24T11:23:58.919815+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:23:58.919815+0000","last_undegraded":"2024-03-24T11:23:58.919815+0000","last_fullsized":"2024-03-24T11:23:58.919815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:44.804631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:44.804631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:02.227249+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":170,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919486+0000","last_change":"2024-03-24T11:21:56.371971+0000","last_active":"2024-03-24T11:23:58.919486+0000","last_peered":"2024-03-24T11:23:58.919486+0000","last_clean":"2024-03-24T11:23:58.919486+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:23:58.919486+0000","last_undegraded":"2024-03-24T11:23:58.919486+0000","last_fullsized":"2024-03-24T11:23:58.919486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:21:56.371834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:21:56.371834+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.938874+0000","last_change":"2024-03-24T11:22:01.345293+0000","last_active":"2024-03-24T11:23:58.938874+0000","last_peered":"2024-03-24T11:23:58.938874+0000","last_clean":"2024-03-24T11:23:58.938874+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:23:58.938874+0000","last_undegraded":"2024-03-24T11:23:58.938874+0000","last_fullsized":"2024-03-24T11:23:58.938874+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:01.345187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:01.345187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925843+0000","last_change":"2024-03-24T11:23:34.136574+0000","last_active":"2024-03-24T11:23:58.925843+0000","last_peered":"2024-03-24T11:23:58.925843+0000","last_clean":"2024-03-24T11:23:58.925843+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:23:58.925843+0000","last_undegraded":"2024-03-24T11:23:58.925843+0000","last_fullsized":"2024-03-24T11:23:58.925843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:34.136448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:34.136448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:51.778171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925851+0000","last_change":"2024-03-24T11:22:08.960884+0000","last_active":"2024-03-24T11:23:58.925851+0000","last_peered":"2024-03-24T11:23:58.925851+0000","last_clean":"2024-03-24T11:23:58.925851+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:23:58.925851+0000","last_undegraded":"2024-03-24T11:23:58.925851+0000","last_fullsized":"2024-03-24T11:23:58.925851+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:08.960803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:08.960803+0000","last_clean_scrub_stamp":"2024-03-24T11:22:08.960803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":169,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.912768+0000","last_change":"2024-03-24T11:23:03.073197+0000","last_active":"2024-03-24T11:23:58.912768+0000","last_peered":"2024-03-24T11:23:58.912768+0000","last_clean":"2024-03-24T11:23:58.912768+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:23:58.912768+0000","last_undegraded":"2024-03-24T11:23:58.912768+0000","last_fullsized":"2024-03-24T11:23:58.912768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:23:03.073053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:03.073053+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:15.313007+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925387+0000","last_change":"2024-03-24T11:22:46.931694+0000","last_active":"2024-03-24T11:23:58.925387+0000","last_peered":"2024-03-24T11:23:58.925387+0000","last_clean":"2024-03-24T11:23:58.925387+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:23:58.925387+0000","last_undegraded":"2024-03-24T11:23:58.925387+0000","last_fullsized":"2024-03-24T11:23:58.925387+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:46.931568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:46.931568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920262+0000","last_change":"2024-03-24T11:22:52.219480+0000","last_active":"2024-03-24T11:23:58.920262+0000","last_peered":"2024-03-24T11:23:58.920262+0000","last_clean":"2024-03-24T11:23:58.920262+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:23:58.920262+0000","last_undegraded":"2024-03-24T11:23:58.920262+0000","last_fullsized":"2024-03-24T11:23:58.920262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:52.219382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:22:52.219382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:04.334219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925658+0000","last_change":"2024-03-24T11:22:06.942899+0000","last_active":"2024-03-24T11:23:58.925658+0000","last_peered":"2024-03-24T11:23:58.925658+0000","last_clean":"2024-03-24T11:23:58.925658+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:23:58.925658+0000","last_undegraded":"2024-03-24T11:23:58.925658+0000","last_fullsized":"2024-03-24T11:23:58.925658+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:06.942785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:22:06.942785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":163,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.912538+0000","last_change":"2024-03-24T11:22:47.267274+0000","last_active":"2024-03-24T11:23:58.912538+0000","last_peered":"2024-03-24T11:23:58.912538+0000","last_clean":"2024-03-24T11:23:58.912538+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:23:58.912538+0000","last_undegraded":"2024-03-24T11:23:58.912538+0000","last_fullsized":"2024-03-24T11:23:58.912538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-24T11:22:47.267120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:47.267120+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914026+0000","last_change":"2024-03-24T11:23:42.101901+0000","last_active":"2024-03-24T11:23:58.914026+0000","last_peered":"2024-03-24T11:23:58.914026+0000","last_clean":"2024-03-24T11:23:58.914026+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:23:58.914026+0000","last_undegraded":"2024-03-24T11:23:58.914026+0000","last_fullsized":"2024-03-24T11:23:58.914026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:42.101792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:42.101792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.152660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.938396+0000","last_change":"2024-03-24T11:23:48.743082+0000","last_active":"2024-03-24T11:23:58.938396+0000","last_peered":"2024-03-24T11:23:58.938396+0000","last_clean":"2024-03-24T11:23:58.938396+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:23:58.938396+0000","last_undegraded":"2024-03-24T11:23:58.938396+0000","last_fullsized":"2024-03-24T11:23:58.938396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:48.742942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:23:48.742942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:08.075374+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.942928+0000","last_change":"2024-03-24T11:22:04.321327+0000","last_active":"2024-03-24T11:23:58.942928+0000","last_peered":"2024-03-24T11:23:58.942928+0000","last_clean":"2024-03-24T11:23:58.942928+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:23:58.942928+0000","last_undegraded":"2024-03-24T11:23:58.942928+0000","last_fullsized":"2024-03-24T11:23:58.942928+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:04.321207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:04.321207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":186,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.926951+0000","last_change":"2024-03-24T11:23:44.878595+0000","last_active":"2024-03-24T11:23:58.926951+0000","last_peered":"2024-03-24T11:23:58.926951+0000","last_clean":"2024-03-24T11:23:58.926951+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:23:58.926951+0000","last_undegraded":"2024-03-24T11:23:58.926951+0000","last_fullsized":"2024-03-24T11:23:58.926951+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:23:44.878429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:44.878429+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:08.355160+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":137,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.924348+0000","last_change":"2024-03-24T11:20:45.487967+0000","last_active":"2024-03-24T11:23:58.924348+0000","last_peered":"2024-03-24T11:23:58.924348+0000","last_clean":"2024-03-24T11:23:58.924348+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:23:58.924348+0000","last_undegraded":"2024-03-24T11:23:58.924348+0000","last_fullsized":"2024-03-24T11:23:58.924348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:45.487868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:20:45.487868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.917183+0000","last_change":"2024-03-24T11:23:43.052281+0000","last_active":"2024-03-24T11:23:58.917183+0000","last_peered":"2024-03-24T11:23:58.917183+0000","last_clean":"2024-03-24T11:23:58.917183+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:23:58.917183+0000","last_undegraded":"2024-03-24T11:23:58.917183+0000","last_fullsized":"2024-03-24T11:23:58.917183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:43.052178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:43.052178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:53.796306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.912787+0000","last_change":"2024-03-24T11:23:02.090932+0000","last_active":"2024-03-24T11:23:58.912787+0000","last_peered":"2024-03-24T11:23:58.912787+0000","last_clean":"2024-03-24T11:23:58.912787+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:23:58.912787+0000","last_undegraded":"2024-03-24T11:23:58.912787+0000","last_fullsized":"2024-03-24T11:23:58.912787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:02.090827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:02.090827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:16.654392+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":168,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919269+0000","last_change":"2024-03-24T11:21:46.227054+0000","last_active":"2024-03-24T11:23:58.919269+0000","last_peered":"2024-03-24T11:23:58.919269+0000","last_clean":"2024-03-24T11:23:58.919269+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:23:58.919269+0000","last_undegraded":"2024-03-24T11:23:58.919269+0000","last_fullsized":"2024-03-24T11:23:58.919269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:21:46.226921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:21:46.226921+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":147,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920871+0000","last_change":"2024-03-24T11:22:06.191366+0000","last_active":"2024-03-24T11:23:58.920871+0000","last_peered":"2024-03-24T11:23:58.920871+0000","last_clean":"2024-03-24T11:23:58.920871+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:23:58.920871+0000","last_undegraded":"2024-03-24T11:23:58.920871+0000","last_fullsized":"2024-03-24T11:23:58.920871+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:22:06.191189+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_clean_scrub_stamp":"2024-03-24T11:22:06.191189+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":234,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.918298+0000","last_change":"2024-03-24T11:23:53.129532+0000","last_active":"2024-03-24T11:23:58.918298+0000","last_peered":"2024-03-24T11:23:58.918298+0000","last_clean":"2024-03-24T11:23:58.918298+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:23:58.918298+0000","last_undegraded":"2024-03-24T11:23:58.918298+0000","last_fullsized":"2024-03-24T11:23:58.918298+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:23:53.129429+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:23:53.129429+0000","last_clean_scrub_stamp":"2024-03-24T11:23:53.129429+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:58.506491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":166,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.924401+0000","last_change":"2024-03-24T11:22:47.648652+0000","last_active":"2024-03-24T11:23:58.924401+0000","last_peered":"2024-03-24T11:23:58.924401+0000","last_clean":"2024-03-24T11:23:58.924401+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:23:58.924401+0000","last_undegraded":"2024-03-24T11:23:58.924401+0000","last_fullsized":"2024-03-24T11:23:58.924401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:22:47.648516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:47.648516+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:16.763189+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":176,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.966934+0000","last_change":"2024-03-24T11:23:05.730005+0000","last_active":"2024-03-24T11:23:58.966934+0000","last_peered":"2024-03-24T11:23:58.966934+0000","last_clean":"2024-03-24T11:23:58.966934+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:23:58.966934+0000","last_undegraded":"2024-03-24T11:23:58.966934+0000","last_fullsized":"2024-03-24T11:23:58.966934+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:23:05.729834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:05.729834+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:06.291793+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.939048+0000","last_change":"2024-03-24T11:23:58.788590+0000","last_active":"2024-03-24T11:23:58.939048+0000","last_peered":"2024-03-24T11:23:58.939048+0000","last_clean":"2024-03-24T11:23:58.939048+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:23:58.939048+0000","last_undegraded":"2024-03-24T11:23:58.939048+0000","last_fullsized":"2024-03-24T11:23:58.939048+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:58.788469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:58.788469+0000","last_clean_scrub_stamp":"2024-03-24T11:23:58.788469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:22.488204+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.964342+0000","last_change":"2024-03-24T11:23:51.912789+0000","last_active":"2024-03-24T11:23:58.964342+0000","last_peered":"2024-03-24T11:23:58.964342+0000","last_clean":"2024-03-24T11:23:58.964342+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:23:58.964342+0000","last_undegraded":"2024-03-24T11:23:58.964342+0000","last_fullsized":"2024-03-24T11:23:58.964342+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:51.912682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:51.912682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:01.869066+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919706+0000","last_change":"2024-03-24T11:22:16.711169+0000","last_active":"2024-03-24T11:23:58.919706+0000","last_peered":"2024-03-24T11:23:58.919706+0000","last_clean":"2024-03-24T11:23:58.919706+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:23:58.919706+0000","last_undegraded":"2024-03-24T11:23:58.919706+0000","last_fullsized":"2024-03-24T11:23:58.919706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:16.710808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:16.710808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":185,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.918207+0000","last_change":"2024-03-24T11:23:06.264074+0000","last_active":"2024-03-24T11:23:58.918207+0000","last_peered":"2024-03-24T11:23:58.918207+0000","last_clean":"2024-03-24T11:23:58.918207+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:23:58.918207+0000","last_undegraded":"2024-03-24T11:23:58.918207+0000","last_fullsized":"2024-03-24T11:23:58.918207+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:23:06.263941+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:23.757510+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920102+0000","last_change":"2024-03-24T11:21:50.099971+0000","last_active":"2024-03-24T11:23:58.920102+0000","last_peered":"2024-03-24T11:23:58.920102+0000","last_clean":"2024-03-24T11:23:58.920102+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:23:58.920102+0000","last_undegraded":"2024-03-24T11:23:58.920102+0000","last_fullsized":"2024-03-24T11:23:58.920102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:50.099842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:21:50.099842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.917831+0000","last_change":"2024-03-24T11:23:55.047105+0000","last_active":"2024-03-24T11:23:58.917831+0000","last_peered":"2024-03-24T11:23:58.917831+0000","last_clean":"2024-03-24T11:23:58.917831+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:23:58.917831+0000","last_undegraded":"2024-03-24T11:23:58.917831+0000","last_fullsized":"2024-03-24T11:23:58.917831+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:55.046997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:55.046997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:59.395282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":165,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.950396+0000","last_change":"2024-03-24T11:23:50.889638+0000","last_active":"2024-03-24T11:23:58.950396+0000","last_peered":"2024-03-24T11:23:58.950396+0000","last_clean":"2024-03-24T11:23:58.950396+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:23:58.950396+0000","last_undegraded":"2024-03-24T11:23:58.950396+0000","last_fullsized":"2024-03-24T11:23:58.950396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:50.889531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:50.889531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:55.943703+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"87'6","reported_seq":161,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.931494+0000","last_change":"2024-03-24T11:22:58.803535+0000","last_active":"2024-03-24T11:23:58.931494+0000","last_peered":"2024-03-24T11:23:58.931494+0000","last_clean":"2024-03-24T11:23:58.931494+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:23:58.931494+0000","last_undegraded":"2024-03-24T11:23:58.931494+0000","last_fullsized":"2024-03-24T11:23:58.931494+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'6","last_scrub_stamp":"2024-03-24T11:22:58.803377+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:22:58.803377+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:22.994062+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.923591+0000","last_change":"2024-03-24T11:21:42.205071+0000","last_active":"2024-03-24T11:23:58.923591+0000","last_peered":"2024-03-24T11:23:58.923591+0000","last_clean":"2024-03-24T11:23:58.923591+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:23:58.923591+0000","last_undegraded":"2024-03-24T11:23:58.923591+0000","last_fullsized":"2024-03-24T11:23:58.923591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:42.202202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:21:42.202202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921685+0000","last_change":"2024-03-24T11:22:54.242371+0000","last_active":"2024-03-24T11:23:58.921685+0000","last_peered":"2024-03-24T11:23:58.921685+0000","last_clean":"2024-03-24T11:23:58.921685+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:23:58.921685+0000","last_undegraded":"2024-03-24T11:23:58.921685+0000","last_fullsized":"2024-03-24T11:23:58.921685+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:54.242252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:54.242252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:18.184119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.926343+0000","last_change":"2024-03-24T11:22:02.947840+0000","last_active":"2024-03-24T11:23:58.926343+0000","last_peered":"2024-03-24T11:23:58.926343+0000","last_clean":"2024-03-24T11:23:58.926343+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:23:58.926343+0000","last_undegraded":"2024-03-24T11:23:58.926343+0000","last_fullsized":"2024-03-24T11:23:58.926343+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:02.947712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:02.947712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":164,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919383+0000","last_change":"2024-03-24T11:20:42.451337+0000","last_active":"2024-03-24T11:23:58.919383+0000","last_peered":"2024-03-24T11:23:58.919383+0000","last_clean":"2024-03-24T11:23:58.919383+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:23:58.919383+0000","last_undegraded":"2024-03-24T11:23:58.919383+0000","last_fullsized":"2024-03-24T11:23:58.919383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:20:42.451248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:20:42.451248+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.938541+0000","last_change":"2024-03-24T11:22:10.270291+0000","last_active":"2024-03-24T11:23:58.938541+0000","last_peered":"2024-03-24T11:23:58.938541+0000","last_clean":"2024-03-24T11:23:58.938541+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:23:58.938541+0000","last_undegraded":"2024-03-24T11:23:58.938541+0000","last_fullsized":"2024-03-24T11:23:58.938541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:10.270179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:10.270179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919453+0000","last_change":"2024-03-24T11:22:52.829399+0000","last_active":"2024-03-24T11:23:58.919453+0000","last_peered":"2024-03-24T11:23:58.919453+0000","last_clean":"2024-03-24T11:23:58.919453+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:23:58.919453+0000","last_undegraded":"2024-03-24T11:23:58.919453+0000","last_fullsized":"2024-03-24T11:23:58.919453+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:52.829170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:52.829170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:10.760699+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920915+0000","last_change":"2024-03-24T11:23:01.249102+0000","last_active":"2024-03-24T11:23:58.920915+0000","last_peered":"2024-03-24T11:23:58.920915+0000","last_clean":"2024-03-24T11:23:58.920915+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:23:58.920915+0000","last_undegraded":"2024-03-24T11:23:58.920915+0000","last_fullsized":"2024-03-24T11:23:58.920915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:01.248993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:01.248993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:12.573856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":177,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.922041+0000","last_change":"2024-03-24T11:23:42.828497+0000","last_active":"2024-03-24T11:23:58.922041+0000","last_peered":"2024-03-24T11:23:58.922041+0000","last_clean":"2024-03-24T11:23:58.922041+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:23:58.922041+0000","last_undegraded":"2024-03-24T11:23:58.922041+0000","last_fullsized":"2024-03-24T11:23:58.922041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:23:42.828353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:42.828353+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:10.358047+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919048+0000","last_change":"2024-03-24T11:22:48.823812+0000","last_active":"2024-03-24T11:23:58.919048+0000","last_peered":"2024-03-24T11:23:58.919048+0000","last_clean":"2024-03-24T11:23:58.919048+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:23:58.919048+0000","last_undegraded":"2024-03-24T11:23:58.919048+0000","last_fullsized":"2024-03-24T11:23:58.919048+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:48.823575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:48.823575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921784+0000","last_change":"2024-03-24T11:23:01.693876+0000","last_active":"2024-03-24T11:23:58.921784+0000","last_peered":"2024-03-24T11:23:58.921784+0000","last_clean":"2024-03-24T11:23:58.921784+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:23:58.921784+0000","last_undegraded":"2024-03-24T11:23:58.921784+0000","last_fullsized":"2024-03-24T11:23:58.921784+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:01.693561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:01.693561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:25.497839+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.916952+0000","last_change":"2024-03-24T11:21:58.206780+0000","last_active":"2024-03-24T11:23:58.916952+0000","last_peered":"2024-03-24T11:23:58.916952+0000","last_clean":"2024-03-24T11:23:58.916952+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:23:58.916952+0000","last_undegraded":"2024-03-24T11:23:58.916952+0000","last_fullsized":"2024-03-24T11:23:58.916952+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:58.206655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:21:58.206655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"129'16","reported_seq":185,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914240+0000","last_change":"2024-03-24T11:23:52.069885+0000","last_active":"2024-03-24T11:23:58.914240+0000","last_peered":"2024-03-24T11:23:58.914240+0000","last_clean":"2024-03-24T11:23:58.914240+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:23:58.914240+0000","last_undegraded":"2024-03-24T11:23:58.914240+0000","last_fullsized":"2024-03-24T11:23:58.914240+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'16","last_scrub_stamp":"2024-03-24T11:23:52.069733+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:23:52.069733+0000","last_clean_scrub_stamp":"2024-03-24T11:23:52.069733+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:52.541101+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.952360+0000","last_change":"2024-03-24T11:22:59.527137+0000","last_active":"2024-03-24T11:23:58.952360+0000","last_peered":"2024-03-24T11:23:58.952360+0000","last_clean":"2024-03-24T11:23:58.952360+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:23:58.952360+0000","last_undegraded":"2024-03-24T11:23:58.952360+0000","last_fullsized":"2024-03-24T11:23:58.952360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:59.526504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:59.526504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:28.036366+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":227,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925902+0000","last_change":"2024-03-24T11:23:10.020542+0000","last_active":"2024-03-24T11:23:58.925902+0000","last_peered":"2024-03-24T11:23:58.925902+0000","last_clean":"2024-03-24T11:23:58.925902+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:23:58.925902+0000","last_undegraded":"2024-03-24T11:23:58.925902+0000","last_fullsized":"2024-03-24T11:23:58.925902+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:23:10.020395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:10.020395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:26.814817+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920350+0000","last_change":"2024-03-24T11:22:16.067774+0000","last_active":"2024-03-24T11:23:58.920350+0000","last_peered":"2024-03-24T11:23:58.920350+0000","last_clean":"2024-03-24T11:23:58.920350+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:23:58.920350+0000","last_undegraded":"2024-03-24T11:23:58.920350+0000","last_fullsized":"2024-03-24T11:23:58.920350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:16.067640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:16.067640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":168,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921581+0000","last_change":"2024-03-24T11:23:48.940813+0000","last_active":"2024-03-24T11:23:58.921581+0000","last_peered":"2024-03-24T11:23:58.921581+0000","last_clean":"2024-03-24T11:23:58.921581+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:23:58.921581+0000","last_undegraded":"2024-03-24T11:23:58.921581+0000","last_fullsized":"2024-03-24T11:23:58.921581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:23:48.940558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:48.940558+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:49.447199+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914142+0000","last_change":"2024-03-24T11:22:51.126800+0000","last_active":"2024-03-24T11:23:58.914142+0000","last_peered":"2024-03-24T11:23:58.914142+0000","last_clean":"2024-03-24T11:23:58.914142+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:23:58.914142+0000","last_undegraded":"2024-03-24T11:23:58.914142+0000","last_fullsized":"2024-03-24T11:23:58.914142+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:51.126702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:51.126702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":219,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.923837+0000","last_change":"2024-03-24T11:22:07.493621+0000","last_active":"2024-03-24T11:23:58.923837+0000","last_peered":"2024-03-24T11:23:58.923837+0000","last_clean":"2024-03-24T11:23:58.923837+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:23:58.923837+0000","last_undegraded":"2024-03-24T11:23:58.923837+0000","last_fullsized":"2024-03-24T11:23:58.923837+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:22:07.493470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:07.493470+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.943247+0000","last_change":"2024-03-24T11:21:58.307086+0000","last_active":"2024-03-24T11:23:58.943247+0000","last_peered":"2024-03-24T11:23:58.943247+0000","last_clean":"2024-03-24T11:23:58.943247+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:23:58.943247+0000","last_undegraded":"2024-03-24T11:23:58.943247+0000","last_fullsized":"2024-03-24T11:23:58.943247+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:58.306953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:21:58.306953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":177,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925965+0000","last_change":"2024-03-24T11:23:44.172720+0000","last_active":"2024-03-24T11:23:58.925965+0000","last_peered":"2024-03-24T11:23:58.925965+0000","last_clean":"2024-03-24T11:23:58.925965+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:23:58.925965+0000","last_undegraded":"2024-03-24T11:23:58.925965+0000","last_fullsized":"2024-03-24T11:23:58.925965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:23:44.172580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:44.172580+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:10.904357+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.927286+0000","last_change":"2024-03-24T11:23:43.922804+0000","last_active":"2024-03-24T11:23:58.927286+0000","last_peered":"2024-03-24T11:23:58.927286+0000","last_clean":"2024-03-24T11:23:58.927286+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:23:58.927286+0000","last_undegraded":"2024-03-24T11:23:58.927286+0000","last_fullsized":"2024-03-24T11:23:58.927286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:43.922696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:43.922696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:47.029782+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.930141+0000","last_change":"2024-03-24T11:23:02.865036+0000","last_active":"2024-03-24T11:23:58.930141+0000","last_peered":"2024-03-24T11:23:58.930141+0000","last_clean":"2024-03-24T11:23:58.930141+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:23:58.930141+0000","last_undegraded":"2024-03-24T11:23:58.930141+0000","last_fullsized":"2024-03-24T11:23:58.930141+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:02.864883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:02.864883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:07.646895+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920191+0000","last_change":"2024-03-24T11:21:47.000319+0000","last_active":"2024-03-24T11:23:58.920191+0000","last_peered":"2024-03-24T11:23:58.920191+0000","last_clean":"2024-03-24T11:23:58.920191+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:23:58.920191+0000","last_undegraded":"2024-03-24T11:23:58.920191+0000","last_fullsized":"2024-03-24T11:23:58.920191+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:47.000205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:21:47.000205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":181,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.964204+0000","last_change":"2024-03-24T11:23:07.072196+0000","last_active":"2024-03-24T11:23:58.964204+0000","last_peered":"2024-03-24T11:23:58.964204+0000","last_clean":"2024-03-24T11:23:58.964204+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:23:58.964204+0000","last_undegraded":"2024-03-24T11:23:58.964204+0000","last_fullsized":"2024-03-24T11:23:58.964204+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:23:07.072062+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:23:07.072062+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:16.580901+0000","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920821+0000","last_change":"2024-03-24T11:23:35.364234+0000","last_active":"2024-03-24T11:23:58.920821+0000","last_peered":"2024-03-24T11:23:58.920821+0000","last_clean":"2024-03-24T11:23:58.920821+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:23:58.920821+0000","last_undegraded":"2024-03-24T11:23:58.920821+0000","last_fullsized":"2024-03-24T11:23:58.920821+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:35.364151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:35.364151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:48.431774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.926342+0000","last_change":"2024-03-24T11:23:36.194234+0000","last_active":"2024-03-24T11:23:58.926342+0000","last_peered":"2024-03-24T11:23:58.926342+0000","last_clean":"2024-03-24T11:23:58.926342+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:23:58.926342+0000","last_undegraded":"2024-03-24T11:23:58.926342+0000","last_fullsized":"2024-03-24T11:23:58.926342+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:36.193867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:36.193867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:36.792165+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":165,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.916627+0000","last_change":"2024-03-24T11:22:53.447680+0000","last_active":"2024-03-24T11:23:58.916627+0000","last_peered":"2024-03-24T11:23:58.916627+0000","last_clean":"2024-03-24T11:23:58.916627+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:23:58.916627+0000","last_undegraded":"2024-03-24T11:23:58.916627+0000","last_fullsized":"2024-03-24T11:23:58.916627+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:53.447553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:53.447553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:23:59.994157+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":162,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.930219+0000","last_change":"2024-03-24T11:23:38.927421+0000","last_active":"2024-03-24T11:23:58.930219+0000","last_peered":"2024-03-24T11:23:58.930219+0000","last_clean":"2024-03-24T11:23:58.930219+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:23:58.930219+0000","last_undegraded":"2024-03-24T11:23:58.930219+0000","last_fullsized":"2024-03-24T11:23:58.930219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-03-24T11:23:38.927200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:38.927200+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:05.038148+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"130'12575","reported_seq":12734,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.197488+0000","last_change":"2024-03-24T11:22:17.723626+0000","last_active":"2024-03-24T11:24:03.197488+0000","last_peered":"2024-03-24T11:24:03.197488+0000","last_clean":"2024-03-24T11:24:03.197488+0000","last_became_active":"2024-03-24T11:20:52.242286+0000","last_became_peered":"2024-03-24T11:20:52.242286+0000","last_unstale":"2024-03-24T11:24:03.197488+0000","last_undegraded":"2024-03-24T11:24:03.197488+0000","last_fullsized":"2024-03-24T11:24:03.197488+0000","mapping_epoch":92,"log_start":"117'7500","ondisk_log_start":"117'7500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6268","last_scrub_stamp":"2024-03-24T11:22:17.723442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:22:17.723442+0000","objects_scrubbed":6213,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:05.837998+0000","scrub_duration":2038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158007296,"num_objects":2411,"num_object_clones":0,"num_object_copies":7233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12575,"num_write_kb":479552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925694+0000","last_change":"2024-03-24T11:23:11.030261+0000","last_active":"2024-03-24T11:23:58.925694+0000","last_peered":"2024-03-24T11:23:58.925694+0000","last_clean":"2024-03-24T11:23:58.925694+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:23:58.925694+0000","last_undegraded":"2024-03-24T11:23:58.925694+0000","last_fullsized":"2024-03-24T11:23:58.925694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:11.030185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:11.030185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:23.326423+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":169,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921253+0000","last_change":"2024-03-24T11:21:59.221535+0000","last_active":"2024-03-24T11:23:58.921253+0000","last_peered":"2024-03-24T11:23:58.921253+0000","last_clean":"2024-03-24T11:23:58.921253+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:23:58.921253+0000","last_undegraded":"2024-03-24T11:23:58.921253+0000","last_fullsized":"2024-03-24T11:23:58.921253+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-24T11:21:59.221397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:21:59.221397+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"129'22","reported_seq":193,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925457+0000","last_change":"2024-03-24T11:23:54.165421+0000","last_active":"2024-03-24T11:23:58.925457+0000","last_peered":"2024-03-24T11:23:58.925457+0000","last_clean":"2024-03-24T11:23:58.925457+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:23:58.925457+0000","last_undegraded":"2024-03-24T11:23:58.925457+0000","last_fullsized":"2024-03-24T11:23:58.925457+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'22","last_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:23:54.165277+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:07.560051+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":219,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920339+0000","last_change":"2024-03-24T11:21:46.036047+0000","last_active":"2024-03-24T11:23:58.920339+0000","last_peered":"2024-03-24T11:23:58.920339+0000","last_clean":"2024-03-24T11:23:58.920339+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:23:58.920339+0000","last_undegraded":"2024-03-24T11:23:58.920339+0000","last_fullsized":"2024-03-24T11:23:58.920339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:21:46.035922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:21:46.035922+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"130'12370","reported_seq":12457,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.421052+0000","last_change":"2024-03-24T11:20:52.244964+0000","last_active":"2024-03-24T11:24:03.421052+0000","last_peered":"2024-03-24T11:24:03.421052+0000","last_clean":"2024-03-24T11:24:03.421052+0000","last_became_active":"2024-03-24T11:20:52.244672+0000","last_became_peered":"2024-03-24T11:20:52.244672+0000","last_unstale":"2024-03-24T11:24:03.421052+0000","last_undegraded":"2024-03-24T11:24:03.421052+0000","last_fullsized":"2024-03-24T11:24:03.421052+0000","mapping_epoch":92,"log_start":"117'7700","ondisk_log_start":"117'7700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:20:51.218936+0000","objects_scrubbed":0,"log_size":4670,"log_dups_size":0,"ondisk_log_size":4670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:04.862559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151257088,"num_objects":2308,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12370,"num_write_kb":469696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.923672+0000","last_change":"2024-03-24T11:23:41.875822+0000","last_active":"2024-03-24T11:23:58.923672+0000","last_peered":"2024-03-24T11:23:58.923672+0000","last_clean":"2024-03-24T11:23:58.923672+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:23:58.923672+0000","last_undegraded":"2024-03-24T11:23:58.923672+0000","last_fullsized":"2024-03-24T11:23:58.923672+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:41.875525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:41.875525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:51.278740+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921149+0000","last_change":"2024-03-24T11:22:51.217432+0000","last_active":"2024-03-24T11:23:58.921149+0000","last_peered":"2024-03-24T11:23:58.921149+0000","last_clean":"2024-03-24T11:23:58.921149+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:23:58.921149+0000","last_undegraded":"2024-03-24T11:23:58.921149+0000","last_fullsized":"2024-03-24T11:23:58.921149+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:51.217352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:22:51.217352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:10.380507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":176,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914672+0000","last_change":"2024-03-24T11:22:10.312507+0000","last_active":"2024-03-24T11:23:58.914672+0000","last_peered":"2024-03-24T11:23:58.914672+0000","last_clean":"2024-03-24T11:23:58.914672+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:23:58.914672+0000","last_undegraded":"2024-03-24T11:23:58.914672+0000","last_fullsized":"2024-03-24T11:23:58.914672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:22:10.312361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:10.312361+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.932251+0000","last_change":"2024-03-24T11:23:41.911951+0000","last_active":"2024-03-24T11:23:58.932251+0000","last_peered":"2024-03-24T11:23:58.932251+0000","last_clean":"2024-03-24T11:23:58.932251+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:23:58.932251+0000","last_undegraded":"2024-03-24T11:23:58.932251+0000","last_fullsized":"2024-03-24T11:23:58.932251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:41.911667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:41.911667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:53.333469+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"130'12477","reported_seq":12631,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.616143+0000","last_change":"2024-03-24T11:22:14.759975+0000","last_active":"2024-03-24T11:24:03.616143+0000","last_peered":"2024-03-24T11:24:03.616143+0000","last_clean":"2024-03-24T11:24:03.616143+0000","last_became_active":"2024-03-24T11:20:52.239023+0000","last_became_peered":"2024-03-24T11:20:52.239023+0000","last_unstale":"2024-03-24T11:24:03.616143+0000","last_undegraded":"2024-03-24T11:24:03.616143+0000","last_fullsized":"2024-03-24T11:24:03.616143+0000","mapping_epoch":92,"log_start":"115'6900","ondisk_log_start":"115'6900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'6034","last_scrub_stamp":"2024-03-24T11:22:14.759713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:22:14.759713+0000","objects_scrubbed":5972,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149225472,"num_objects":2277,"num_object_clones":0,"num_object_copies":6831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":472128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921089+0000","last_change":"2024-03-24T11:22:59.216187+0000","last_active":"2024-03-24T11:23:58.921089+0000","last_peered":"2024-03-24T11:23:58.921089+0000","last_clean":"2024-03-24T11:23:58.921089+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:23:58.921089+0000","last_undegraded":"2024-03-24T11:23:58.921089+0000","last_fullsized":"2024-03-24T11:23:58.921089+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:59.216075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:59.216075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:07.899095+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.927592+0000","last_change":"2024-03-24T11:23:48.839167+0000","last_active":"2024-03-24T11:23:58.927592+0000","last_peered":"2024-03-24T11:23:58.927592+0000","last_clean":"2024-03-24T11:23:58.927592+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:23:58.927592+0000","last_undegraded":"2024-03-24T11:23:58.927592+0000","last_fullsized":"2024-03-24T11:23:58.927592+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:23:48.839016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:04.535882+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921030+0000","last_change":"2024-03-24T11:21:43.983111+0000","last_active":"2024-03-24T11:23:58.921030+0000","last_peered":"2024-03-24T11:23:58.921030+0000","last_clean":"2024-03-24T11:23:58.921030+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:23:58.921030+0000","last_undegraded":"2024-03-24T11:23:58.921030+0000","last_fullsized":"2024-03-24T11:23:58.921030+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:43.982992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:21:43.982992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"88'28","reported_seq":201,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.917038+0000","last_change":"2024-03-24T11:23:05.239882+0000","last_active":"2024-03-24T11:23:58.917038+0000","last_peered":"2024-03-24T11:23:58.917038+0000","last_clean":"2024-03-24T11:23:58.917038+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:23:58.917038+0000","last_undegraded":"2024-03-24T11:23:58.917038+0000","last_fullsized":"2024-03-24T11:23:58.917038+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'28","last_scrub_stamp":"2024-03-24T11:23:05.239780+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:23:05.239780+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:17.897916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"130'12454","reported_seq":12642,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.248291+0000","last_change":"2024-03-24T11:23:54.076703+0000","last_active":"2024-03-24T11:24:03.248291+0000","last_peered":"2024-03-24T11:24:03.248291+0000","last_clean":"2024-03-24T11:24:03.248291+0000","last_became_active":"2024-03-24T11:20:52.242525+0000","last_became_peered":"2024-03-24T11:20:52.242525+0000","last_unstale":"2024-03-24T11:24:03.248291+0000","last_undegraded":"2024-03-24T11:24:03.248291+0000","last_fullsized":"2024-03-24T11:24:03.248291+0000","mapping_epoch":92,"log_start":"110'5600","ondisk_log_start":"110'5600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'11898","last_scrub_stamp":"2024-03-24T11:23:54.076547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:54.076547+0000","objects_scrubbed":2862,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:22.445975+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149159936,"num_objects":2276,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":471360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.943382+0000","last_change":"2024-03-24T11:23:39.866113+0000","last_active":"2024-03-24T11:23:58.943382+0000","last_peered":"2024-03-24T11:23:58.943382+0000","last_clean":"2024-03-24T11:23:58.943382+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:23:58.943382+0000","last_undegraded":"2024-03-24T11:23:58.943382+0000","last_fullsized":"2024-03-24T11:23:58.943382+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:39.859771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:39.859771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:45.421006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919528+0000","last_change":"2024-03-24T11:23:34.845524+0000","last_active":"2024-03-24T11:23:58.919528+0000","last_peered":"2024-03-24T11:23:58.919528+0000","last_clean":"2024-03-24T11:23:58.919528+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:23:58.919528+0000","last_undegraded":"2024-03-24T11:23:58.919528+0000","last_fullsized":"2024-03-24T11:23:58.919528+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:34.845279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:34.845279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:35.027129+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":171,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925716+0000","last_change":"2024-03-24T11:23:42.165766+0000","last_active":"2024-03-24T11:23:58.925716+0000","last_peered":"2024-03-24T11:23:58.925716+0000","last_clean":"2024-03-24T11:23:58.925716+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:23:58.925716+0000","last_undegraded":"2024-03-24T11:23:58.925716+0000","last_fullsized":"2024-03-24T11:23:58.925716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:23:42.165617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:42.165617+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:01.706182+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":177,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.943055+0000","last_change":"2024-03-24T11:23:42.898522+0000","last_active":"2024-03-24T11:23:58.943055+0000","last_peered":"2024-03-24T11:23:58.943055+0000","last_clean":"2024-03-24T11:23:58.943055+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:23:58.943055+0000","last_undegraded":"2024-03-24T11:23:58.943055+0000","last_fullsized":"2024-03-24T11:23:58.943055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:23:42.898393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:42.898393+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:58.174249+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"130'12340","reported_seq":12524,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.084981+0000","last_change":"2024-03-24T11:23:49.162010+0000","last_active":"2024-03-24T11:24:03.084981+0000","last_peered":"2024-03-24T11:24:03.084981+0000","last_clean":"2024-03-24T11:24:03.084981+0000","last_became_active":"2024-03-24T11:20:52.243805+0000","last_became_peered":"2024-03-24T11:20:52.243805+0000","last_unstale":"2024-03-24T11:24:03.084981+0000","last_undegraded":"2024-03-24T11:24:03.084981+0000","last_fullsized":"2024-03-24T11:24:03.084981+0000","mapping_epoch":92,"log_start":"112'6000","ondisk_log_start":"112'6000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'11492","last_scrub_stamp":"2024-03-24T11:23:49.161826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:49.161826+0000","objects_scrubbed":3267,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.621939+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155713536,"num_objects":2376,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":470912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920160+0000","last_change":"2024-03-24T11:23:00.798000+0000","last_active":"2024-03-24T11:23:58.920160+0000","last_peered":"2024-03-24T11:23:58.920160+0000","last_clean":"2024-03-24T11:23:58.920160+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:23:58.920160+0000","last_undegraded":"2024-03-24T11:23:58.920160+0000","last_fullsized":"2024-03-24T11:23:58.920160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:00.797777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:00.797777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:03.797239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":176,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925545+0000","last_change":"2024-03-24T11:23:49.141717+0000","last_active":"2024-03-24T11:23:58.925545+0000","last_peered":"2024-03-24T11:23:58.925545+0000","last_clean":"2024-03-24T11:23:58.925545+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:23:58.925545+0000","last_undegraded":"2024-03-24T11:23:58.925545+0000","last_fullsized":"2024-03-24T11:23:58.925545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:23:49.141571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:23:49.141571+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:05.135458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.912656+0000","last_change":"2024-03-24T11:22:07.275053+0000","last_active":"2024-03-24T11:23:58.912656+0000","last_peered":"2024-03-24T11:23:58.912656+0000","last_clean":"2024-03-24T11:23:58.912656+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:23:58.912656+0000","last_undegraded":"2024-03-24T11:23:58.912656+0000","last_fullsized":"2024-03-24T11:23:58.912656+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:07.274968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:07.274968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":271,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.929681+0000","last_change":"2024-03-24T11:23:39.868223+0000","last_active":"2024-03-24T11:23:58.929681+0000","last_peered":"2024-03-24T11:23:58.929681+0000","last_clean":"2024-03-24T11:23:58.929681+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:23:58.929681+0000","last_undegraded":"2024-03-24T11:23:58.929681+0000","last_fullsized":"2024-03-24T11:23:58.929681+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:23:39.868007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:39.868007+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:03.450248+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.939658+0000","last_change":"2024-03-24T11:22:08.296568+0000","last_active":"2024-03-24T11:23:58.939658+0000","last_peered":"2024-03-24T11:23:58.939658+0000","last_clean":"2024-03-24T11:23:58.939658+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:23:58.939658+0000","last_undegraded":"2024-03-24T11:23:58.939658+0000","last_fullsized":"2024-03-24T11:23:58.939658+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:08.296333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:08.296333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"130'12602","reported_seq":12793,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.604762+0000","last_change":"2024-03-24T11:22:51.145802+0000","last_active":"2024-03-24T11:24:03.604762+0000","last_peered":"2024-03-24T11:24:03.604762+0000","last_clean":"2024-03-24T11:24:03.604762+0000","last_became_active":"2024-03-24T11:20:52.236455+0000","last_became_peered":"2024-03-24T11:20:52.236455+0000","last_unstale":"2024-03-24T11:24:03.604762+0000","last_undegraded":"2024-03-24T11:24:03.604762+0000","last_fullsized":"2024-03-24T11:24:03.604762+0000","mapping_epoch":92,"log_start":"115'7000","ondisk_log_start":"115'7000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8324","last_scrub_stamp":"2024-03-24T11:22:51.145618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:22:51.145618+0000","objects_scrubbed":6724,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:15.602673+0000","scrub_duration":2769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154009600,"num_objects":2350,"num_object_clones":0,"num_object_copies":7050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":478464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921138+0000","last_change":"2024-03-24T11:23:36.381816+0000","last_active":"2024-03-24T11:23:58.921138+0000","last_peered":"2024-03-24T11:23:58.921138+0000","last_clean":"2024-03-24T11:23:58.921138+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:23:58.921138+0000","last_undegraded":"2024-03-24T11:23:58.921138+0000","last_fullsized":"2024-03-24T11:23:58.921138+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:36.381692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:36.381692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:49.291630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":165,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.923989+0000","last_change":"2024-03-24T11:22:50.646492+0000","last_active":"2024-03-24T11:23:58.923989+0000","last_peered":"2024-03-24T11:23:58.923989+0000","last_clean":"2024-03-24T11:23:58.923989+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:23:58.923989+0000","last_undegraded":"2024-03-24T11:23:58.923989+0000","last_fullsized":"2024-03-24T11:23:58.923989+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:50.646374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:50.646374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:14.082868+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":174,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919358+0000","last_change":"2024-03-24T11:23:50.832438+0000","last_active":"2024-03-24T11:23:58.919358+0000","last_peered":"2024-03-24T11:23:58.919358+0000","last_clean":"2024-03-24T11:23:58.919358+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:23:58.919358+0000","last_undegraded":"2024-03-24T11:23:58.919358+0000","last_fullsized":"2024-03-24T11:23:58.919358+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:23:50.832319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:07.534574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":155,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.932732+0000","last_change":"2024-03-24T11:22:59.968589+0000","last_active":"2024-03-24T11:23:58.932732+0000","last_peered":"2024-03-24T11:23:58.932732+0000","last_clean":"2024-03-24T11:23:58.932732+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:23:58.932732+0000","last_undegraded":"2024-03-24T11:23:58.932732+0000","last_fullsized":"2024-03-24T11:23:58.932732+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:22:59.968441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:59.968441+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:11.225040+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.931575+0000","last_change":"2024-03-24T11:23:02.959118+0000","last_active":"2024-03-24T11:23:58.931575+0000","last_peered":"2024-03-24T11:23:58.931575+0000","last_clean":"2024-03-24T11:23:58.931575+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:23:58.931575+0000","last_undegraded":"2024-03-24T11:23:58.931575+0000","last_fullsized":"2024-03-24T11:23:58.931575+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:02.958981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:02.958981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:10.396054+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"130'15259","reported_seq":15499,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.619503+0000","last_change":"2024-03-24T11:23:38.651723+0000","last_active":"2024-03-24T11:24:03.619503+0000","last_peered":"2024-03-24T11:24:03.619503+0000","last_clean":"2024-03-24T11:24:03.619503+0000","last_became_active":"2024-03-24T11:20:52.241516+0000","last_became_peered":"2024-03-24T11:20:52.241516+0000","last_unstale":"2024-03-24T11:24:03.619503+0000","last_undegraded":"2024-03-24T11:24:03.619503+0000","last_fullsized":"2024-03-24T11:24:03.619503+0000","mapping_epoch":92,"log_start":"120'9700","ondisk_log_start":"120'9700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'13669","last_scrub_stamp":"2024-03-24T11:23:38.651413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:38.651413+0000","objects_scrubbed":3968,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:46.366698+0000","scrub_duration":1766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152109056,"num_objects":2321,"num_object_clones":0,"num_object_copies":6963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12807,"num_write_kb":484096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.926392+0000","last_change":"2024-03-24T11:22:58.967932+0000","last_active":"2024-03-24T11:23:58.926392+0000","last_peered":"2024-03-24T11:23:58.926392+0000","last_clean":"2024-03-24T11:23:58.926392+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:23:58.926392+0000","last_undegraded":"2024-03-24T11:23:58.926392+0000","last_fullsized":"2024-03-24T11:23:58.926392+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:58.967806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:58.967806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:02.679066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":410,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921266+0000","last_change":"2024-03-24T11:23:02.255831+0000","last_active":"2024-03-24T11:23:58.921266+0000","last_peered":"2024-03-24T11:23:58.921266+0000","last_clean":"2024-03-24T11:23:58.921266+0000","last_became_active":"2024-03-24T11:18:44.123994+0000","last_became_peered":"2024-03-24T11:18:44.123994+0000","last_unstale":"2024-03-24T11:23:58.921266+0000","last_undegraded":"2024-03-24T11:23:58.921266+0000","last_fullsized":"2024-03-24T11:23:58.921266+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:23:02.255686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:23:02.255686+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:19.821759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.924053+0000","last_change":"2024-03-24T11:22:49.850872+0000","last_active":"2024-03-24T11:23:58.924053+0000","last_peered":"2024-03-24T11:23:58.924053+0000","last_clean":"2024-03-24T11:23:58.924053+0000","last_became_active":"2024-03-24T11:18:53.678099+0000","last_became_peered":"2024-03-24T11:18:53.678099+0000","last_unstale":"2024-03-24T11:23:58.924053+0000","last_undegraded":"2024-03-24T11:23:58.924053+0000","last_fullsized":"2024-03-24T11:23:58.924053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:49.850716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:49.850716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":245,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.923650+0000","last_change":"2024-03-24T11:22:54.394020+0000","last_active":"2024-03-24T11:23:58.923650+0000","last_peered":"2024-03-24T11:23:58.923650+0000","last_clean":"2024-03-24T11:23:58.923650+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:23:58.923650+0000","last_undegraded":"2024-03-24T11:23:58.923650+0000","last_fullsized":"2024-03-24T11:23:58.923650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-24T11:22:54.393824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:54.393824+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:01.122468+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920929+0000","last_change":"2024-03-24T11:23:42.431068+0000","last_active":"2024-03-24T11:23:58.920929+0000","last_peered":"2024-03-24T11:23:58.920929+0000","last_clean":"2024-03-24T11:23:58.920929+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:23:58.920929+0000","last_undegraded":"2024-03-24T11:23:58.920929+0000","last_fullsized":"2024-03-24T11:23:58.920929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:42.430955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:42.430955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:44.369980+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"130'12634","reported_seq":12796,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.037377+0000","last_change":"2024-03-24T11:22:50.928370+0000","last_active":"2024-03-24T11:24:03.037377+0000","last_peered":"2024-03-24T11:24:03.037377+0000","last_clean":"2024-03-24T11:24:03.037377+0000","last_became_active":"2024-03-24T11:20:52.245695+0000","last_became_peered":"2024-03-24T11:20:52.245695+0000","last_unstale":"2024-03-24T11:24:03.037377+0000","last_undegraded":"2024-03-24T11:24:03.037377+0000","last_fullsized":"2024-03-24T11:24:03.037377+0000","mapping_epoch":92,"log_start":"116'7400","ondisk_log_start":"116'7400","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8318","last_scrub_stamp":"2024-03-24T11:22:50.928168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:22:50.928168+0000","objects_scrubbed":6717,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:17.954426+0000","scrub_duration":1735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12634,"num_write_kb":479552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.929397+0000","last_change":"2024-03-24T11:23:54.778403+0000","last_active":"2024-03-24T11:23:58.929397+0000","last_peered":"2024-03-24T11:23:58.929397+0000","last_clean":"2024-03-24T11:23:58.929397+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:23:58.929397+0000","last_undegraded":"2024-03-24T11:23:58.929397+0000","last_fullsized":"2024-03-24T11:23:58.929397+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:54.777433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:54.777433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:01.260383+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":160,"reported_epoch":130,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T11:23:58.953696+0000","last_change":"2024-03-24T11:23:58.936654+0000","last_active":"2024-03-24T11:23:58.953696+0000","last_peered":"2024-03-24T11:23:58.953696+0000","last_clean":"2024-03-24T11:23:58.953696+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:23:58.953696+0000","last_undegraded":"2024-03-24T11:23:58.953696+0000","last_fullsized":"2024-03-24T11:23:58.953696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:56.331779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:21:56.331779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":472,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:01.876031+0000","last_change":"2024-03-24T11:22:05.225473+0000","last_active":"2024-03-24T11:24:01.876031+0000","last_peered":"2024-03-24T11:24:01.876031+0000","last_clean":"2024-03-24T11:24:01.876031+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:24:01.876031+0000","last_undegraded":"2024-03-24T11:24:01.876031+0000","last_fullsized":"2024-03-24T11:24:01.876031+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:22:05.225390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:22:05.225390+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":303,"num_read_kb":298,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":171,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.952463+0000","last_change":"2024-03-24T11:22:13.286541+0000","last_active":"2024-03-24T11:23:58.952463+0000","last_peered":"2024-03-24T11:23:58.952463+0000","last_clean":"2024-03-24T11:23:58.952463+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:23:58.952463+0000","last_undegraded":"2024-03-24T11:23:58.952463+0000","last_fullsized":"2024-03-24T11:23:58.952463+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:22:13.286401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:13.286401+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.926527+0000","last_change":"2024-03-24T11:22:00.948676+0000","last_active":"2024-03-24T11:23:58.926527+0000","last_peered":"2024-03-24T11:23:58.926527+0000","last_clean":"2024-03-24T11:23:58.926527+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:23:58.926527+0000","last_undegraded":"2024-03-24T11:23:58.926527+0000","last_fullsized":"2024-03-24T11:23:58.926527+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:00.948544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:00.948544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"130'12797","reported_seq":12959,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.036669+0000","last_change":"2024-03-24T11:22:51.934584+0000","last_active":"2024-03-24T11:24:03.036669+0000","last_peered":"2024-03-24T11:24:03.036669+0000","last_clean":"2024-03-24T11:24:03.036669+0000","last_became_active":"2024-03-24T11:20:52.241054+0000","last_became_peered":"2024-03-24T11:20:52.241054+0000","last_unstale":"2024-03-24T11:24:03.036669+0000","last_undegraded":"2024-03-24T11:24:03.036669+0000","last_fullsized":"2024-03-24T11:24:03.036669+0000","mapping_epoch":92,"log_start":"115'7000","ondisk_log_start":"115'7000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8506","last_scrub_stamp":"2024-03-24T11:22:51.934469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:22:51.934469+0000","objects_scrubbed":6720,"log_size":5797,"log_dups_size":0,"ondisk_log_size":5797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:04.177018+0000","scrub_duration":1997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":7107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12797,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.924563+0000","last_change":"2024-03-24T11:22:11.763508+0000","last_active":"2024-03-24T11:23:58.924563+0000","last_peered":"2024-03-24T11:23:58.924563+0000","last_clean":"2024-03-24T11:23:58.924563+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:23:58.924563+0000","last_undegraded":"2024-03-24T11:23:58.924563+0000","last_fullsized":"2024-03-24T11:23:58.924563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:11.763275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:11.763275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":165,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.929603+0000","last_change":"2024-03-24T11:23:07.789053+0000","last_active":"2024-03-24T11:23:58.929603+0000","last_peered":"2024-03-24T11:23:58.929603+0000","last_clean":"2024-03-24T11:23:58.929603+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:23:58.929603+0000","last_undegraded":"2024-03-24T11:23:58.929603+0000","last_fullsized":"2024-03-24T11:23:58.929603+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:07.788940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:07.788940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:08.747157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":189,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.942827+0000","last_change":"2024-03-24T11:22:55.379960+0000","last_active":"2024-03-24T11:23:58.942827+0000","last_peered":"2024-03-24T11:23:58.942827+0000","last_clean":"2024-03-24T11:23:58.942827+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:23:58.942827+0000","last_undegraded":"2024-03-24T11:23:58.942827+0000","last_fullsized":"2024-03-24T11:23:58.942827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-24T11:22:55.379808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:55.379808+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:02.940379+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.917123+0000","last_change":"2024-03-24T11:23:09.251948+0000","last_active":"2024-03-24T11:23:58.917123+0000","last_peered":"2024-03-24T11:23:58.917123+0000","last_clean":"2024-03-24T11:23:58.917123+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:23:58.917123+0000","last_undegraded":"2024-03-24T11:23:58.917123+0000","last_fullsized":"2024-03-24T11:23:58.917123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:09.251870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_clean_scrub_stamp":"2024-03-24T11:23:09.251870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:28.114197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"130'12535","reported_seq":12668,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.064280+0000","last_change":"2024-03-24T11:23:39.096268+0000","last_active":"2024-03-24T11:24:03.064280+0000","last_peered":"2024-03-24T11:24:03.064280+0000","last_clean":"2024-03-24T11:24:03.064280+0000","last_became_active":"2024-03-24T11:20:52.240454+0000","last_became_peered":"2024-03-24T11:20:52.240454+0000","last_unstale":"2024-03-24T11:24:03.064280+0000","last_undegraded":"2024-03-24T11:24:03.064280+0000","last_fullsized":"2024-03-24T11:24:03.064280+0000","mapping_epoch":92,"log_start":"117'7700","ondisk_log_start":"117'7700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'11068","last_scrub_stamp":"2024-03-24T11:23:39.096159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:39.096159+0000","objects_scrubbed":3791,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:00.914883+0000","scrub_duration":953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150405120,"num_objects":2295,"num_object_clones":0,"num_object_copies":6885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12535,"num_write_kb":474560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":146,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.938654+0000","last_change":"2024-03-24T11:22:49.433870+0000","last_active":"2024-03-24T11:23:58.938654+0000","last_peered":"2024-03-24T11:23:58.938654+0000","last_clean":"2024-03-24T11:23:58.938654+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:23:58.938654+0000","last_undegraded":"2024-03-24T11:23:58.938654+0000","last_fullsized":"2024-03-24T11:23:58.938654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:22:49.433568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:49.433568+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":165,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925915+0000","last_change":"2024-03-24T11:22:56.013215+0000","last_active":"2024-03-24T11:23:58.925915+0000","last_peered":"2024-03-24T11:23:58.925915+0000","last_clean":"2024-03-24T11:23:58.925915+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:23:58.925915+0000","last_undegraded":"2024-03-24T11:23:58.925915+0000","last_fullsized":"2024-03-24T11:23:58.925915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:56.013141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:56.013141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:07.627921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":176,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921874+0000","last_change":"2024-03-24T11:23:53.770354+0000","last_active":"2024-03-24T11:23:58.921874+0000","last_peered":"2024-03-24T11:23:58.921874+0000","last_clean":"2024-03-24T11:23:58.921874+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:23:58.921874+0000","last_undegraded":"2024-03-24T11:23:58.921874+0000","last_fullsized":"2024-03-24T11:23:58.921874+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:23:53.770204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:53.770204+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:10.181169+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":227,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.943148+0000","last_change":"2024-03-24T11:23:10.665104+0000","last_active":"2024-03-24T11:23:58.943148+0000","last_peered":"2024-03-24T11:23:58.943148+0000","last_clean":"2024-03-24T11:23:58.943148+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:23:58.943148+0000","last_undegraded":"2024-03-24T11:23:58.943148+0000","last_fullsized":"2024-03-24T11:23:58.943148+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:23:10.661989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:10.661989+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:24.317849+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"130'12510","reported_seq":12597,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.376296+0000","last_change":"2024-03-24T11:20:52.244918+0000","last_active":"2024-03-24T11:24:03.376296+0000","last_peered":"2024-03-24T11:24:03.376296+0000","last_clean":"2024-03-24T11:24:03.376296+0000","last_became_active":"2024-03-24T11:20:52.244607+0000","last_became_peered":"2024-03-24T11:20:52.244607+0000","last_unstale":"2024-03-24T11:24:03.376296+0000","last_undegraded":"2024-03-24T11:24:03.376296+0000","last_fullsized":"2024-03-24T11:24:03.376296+0000","mapping_epoch":92,"log_start":"117'7800","ondisk_log_start":"117'7800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:20:51.218936+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:04.862523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154402816,"num_objects":2356,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12510,"num_write_kb":475712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920543+0000","last_change":"2024-03-24T11:22:00.164078+0000","last_active":"2024-03-24T11:23:58.920543+0000","last_peered":"2024-03-24T11:23:58.920543+0000","last_clean":"2024-03-24T11:23:58.920543+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:23:58.920543+0000","last_undegraded":"2024-03-24T11:23:58.920543+0000","last_fullsized":"2024-03-24T11:23:58.920543+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:00.163972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:00.163972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":166,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.960731+0000","last_change":"2024-03-24T11:23:03.699529+0000","last_active":"2024-03-24T11:23:58.960731+0000","last_peered":"2024-03-24T11:23:58.960731+0000","last_clean":"2024-03-24T11:23:58.960731+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:23:58.960731+0000","last_undegraded":"2024-03-24T11:23:58.960731+0000","last_fullsized":"2024-03-24T11:23:58.960731+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:23:03.699359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:03.699359+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:29.870027+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":174,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914562+0000","last_change":"2024-03-24T11:22:46.193806+0000","last_active":"2024-03-24T11:23:58.914562+0000","last_peered":"2024-03-24T11:23:58.914562+0000","last_clean":"2024-03-24T11:23:58.914562+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:23:58.914562+0000","last_undegraded":"2024-03-24T11:23:58.914562+0000","last_fullsized":"2024-03-24T11:23:58.914562+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-03-24T11:22:46.193690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:46.193690+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:02.511822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":211,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.920052+0000","last_change":"2024-03-24T11:21:44.257283+0000","last_active":"2024-03-24T11:23:58.920052+0000","last_peered":"2024-03-24T11:23:58.920052+0000","last_clean":"2024-03-24T11:23:58.920052+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:23:58.920052+0000","last_undegraded":"2024-03-24T11:23:58.920052+0000","last_fullsized":"2024-03-24T11:23:58.920052+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:21:44.257057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:21:44.257057+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"130'12596","reported_seq":12725,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.403318+0000","last_change":"2024-03-24T11:23:44.940404+0000","last_active":"2024-03-24T11:24:03.403318+0000","last_peered":"2024-03-24T11:24:03.403318+0000","last_clean":"2024-03-24T11:24:03.403318+0000","last_became_active":"2024-03-24T11:20:52.243565+0000","last_became_peered":"2024-03-24T11:20:52.243565+0000","last_unstale":"2024-03-24T11:24:03.403318+0000","last_undegraded":"2024-03-24T11:24:03.403318+0000","last_fullsized":"2024-03-24T11:24:03.403318+0000","mapping_epoch":92,"log_start":"118'7900","ondisk_log_start":"118'7900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'11517","last_scrub_stamp":"2024-03-24T11:23:44.940224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:44.940224+0000","objects_scrubbed":3418,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:54.422437+0000","scrub_duration":1145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151126016,"num_objects":2306,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12596,"num_write_kb":476864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.912660+0000","last_change":"2024-03-24T11:23:08.106781+0000","last_active":"2024-03-24T11:23:58.912660+0000","last_peered":"2024-03-24T11:23:58.912660+0000","last_clean":"2024-03-24T11:23:58.912660+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:23:58.912660+0000","last_undegraded":"2024-03-24T11:23:58.912660+0000","last_fullsized":"2024-03-24T11:23:58.912660+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:08.106645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:49.118465+0000","last_clean_scrub_stamp":"2024-03-24T11:23:08.106645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:36.540851+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.929943+0000","last_change":"2024-03-24T11:22:55.832922+0000","last_active":"2024-03-24T11:23:58.929943+0000","last_peered":"2024-03-24T11:23:58.929943+0000","last_clean":"2024-03-24T11:23:58.929943+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:23:58.929943+0000","last_undegraded":"2024-03-24T11:23:58.929943+0000","last_fullsized":"2024-03-24T11:23:58.929943+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:55.832785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:22:55.832785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":177,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919946+0000","last_change":"2024-03-24T11:21:48.273776+0000","last_active":"2024-03-24T11:23:58.919946+0000","last_peered":"2024-03-24T11:23:58.919946+0000","last_clean":"2024-03-24T11:23:58.919946+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:23:58.919946+0000","last_undegraded":"2024-03-24T11:23:58.919946+0000","last_fullsized":"2024-03-24T11:23:58.919946+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-24T11:21:48.273515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:21:48.273515+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914446+0000","last_change":"2024-03-24T11:23:58.123974+0000","last_active":"2024-03-24T11:23:58.914446+0000","last_peered":"2024-03-24T11:23:58.914446+0000","last_clean":"2024-03-24T11:23:58.914446+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:23:58.914446+0000","last_undegraded":"2024-03-24T11:23:58.914446+0000","last_fullsized":"2024-03-24T11:23:58.914446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:58.123848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:58.123848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:04.422450+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"130'12417","reported_seq":12608,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.102624+0000","last_change":"2024-03-24T11:23:42.189190+0000","last_active":"2024-03-24T11:24:03.102624+0000","last_peered":"2024-03-24T11:24:03.102624+0000","last_clean":"2024-03-24T11:24:03.102624+0000","last_became_active":"2024-03-24T11:20:52.240484+0000","last_became_peered":"2024-03-24T11:20:52.240484+0000","last_unstale":"2024-03-24T11:24:03.102624+0000","last_undegraded":"2024-03-24T11:24:03.102624+0000","last_fullsized":"2024-03-24T11:24:03.102624+0000","mapping_epoch":92,"log_start":"113'6100","ondisk_log_start":"113'6100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'11182","last_scrub_stamp":"2024-03-24T11:23:42.188863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:42.188863+0000","objects_scrubbed":3622,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:04.471174+0000","scrub_duration":1008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154337280,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12417,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.925851+0000","last_change":"2024-03-24T11:22:48.991077+0000","last_active":"2024-03-24T11:23:58.925851+0000","last_peered":"2024-03-24T11:23:58.925851+0000","last_clean":"2024-03-24T11:23:58.925851+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:23:58.925851+0000","last_undegraded":"2024-03-24T11:23:58.925851+0000","last_fullsized":"2024-03-24T11:23:58.925851+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:48.990897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:22:48.990897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:15.490583+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":219,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914109+0000","last_change":"2024-03-24T11:22:48.194228+0000","last_active":"2024-03-24T11:23:58.914109+0000","last_peered":"2024-03-24T11:23:58.914109+0000","last_clean":"2024-03-24T11:23:58.914109+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:23:58.914109+0000","last_undegraded":"2024-03-24T11:23:58.914109+0000","last_fullsized":"2024-03-24T11:23:58.914109+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:22:48.194094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:48.194094+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919841+0000","last_change":"2024-03-24T11:22:01.581950+0000","last_active":"2024-03-24T11:23:58.919841+0000","last_peered":"2024-03-24T11:23:58.919841+0000","last_clean":"2024-03-24T11:23:58.919841+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:23:58.919841+0000","last_undegraded":"2024-03-24T11:23:58.919841+0000","last_fullsized":"2024-03-24T11:23:58.919841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:22:01.581731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"89'22","reported_seq":193,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.922128+0000","last_change":"2024-03-24T11:23:56.824853+0000","last_active":"2024-03-24T11:23:58.922128+0000","last_peered":"2024-03-24T11:23:58.922128+0000","last_clean":"2024-03-24T11:23:58.922128+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:23:58.922128+0000","last_undegraded":"2024-03-24T11:23:58.922128+0000","last_fullsized":"2024-03-24T11:23:58.922128+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-03-24T11:23:56.824712+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:23:56.824712+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:01.308655+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"130'12496","reported_seq":12685,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.097611+0000","last_change":"2024-03-24T11:23:56.877951+0000","last_active":"2024-03-24T11:24:03.097611+0000","last_peered":"2024-03-24T11:24:03.097611+0000","last_clean":"2024-03-24T11:24:03.097611+0000","last_became_active":"2024-03-24T11:20:52.243686+0000","last_became_peered":"2024-03-24T11:20:52.243686+0000","last_unstale":"2024-03-24T11:24:03.097611+0000","last_undegraded":"2024-03-24T11:24:03.097611+0000","last_fullsized":"2024-03-24T11:24:03.097611+0000","mapping_epoch":92,"log_start":"113'6200","ondisk_log_start":"113'6200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'12127","last_scrub_stamp":"2024-03-24T11:23:56.877687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:56.877687+0000","objects_scrubbed":2744,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:23.188060+0000","scrub_duration":839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154075160,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12496,"num_write_kb":475073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.921160+0000","last_change":"2024-03-24T11:23:09.925823+0000","last_active":"2024-03-24T11:23:58.921160+0000","last_peered":"2024-03-24T11:23:58.921160+0000","last_clean":"2024-03-24T11:23:58.921160+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:23:58.921160+0000","last_undegraded":"2024-03-24T11:23:58.921160+0000","last_fullsized":"2024-03-24T11:23:58.921160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:09.925715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:09.925715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:29.557084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":149,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.926497+0000","last_change":"2024-03-24T11:22:05.018747+0000","last_active":"2024-03-24T11:23:58.926497+0000","last_peered":"2024-03-24T11:23:58.926497+0000","last_clean":"2024-03-24T11:23:58.926497+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:23:58.926497+0000","last_undegraded":"2024-03-24T11:23:58.926497+0000","last_fullsized":"2024-03-24T11:23:58.926497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:05.018664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:05.018664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914470+0000","last_change":"2024-03-24T11:22:54.193791+0000","last_active":"2024-03-24T11:23:58.914470+0000","last_peered":"2024-03-24T11:23:58.914470+0000","last_clean":"2024-03-24T11:23:58.914470+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:23:58.914470+0000","last_undegraded":"2024-03-24T11:23:58.914470+0000","last_fullsized":"2024-03-24T11:23:58.914470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:54.193661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:54.193661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:21.224227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":180,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.913940+0000","last_change":"2024-03-24T11:23:41.128542+0000","last_active":"2024-03-24T11:23:58.913940+0000","last_peered":"2024-03-24T11:23:58.913940+0000","last_clean":"2024-03-24T11:23:58.913940+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:23:58.913940+0000","last_undegraded":"2024-03-24T11:23:58.913940+0000","last_fullsized":"2024-03-24T11:23:58.913940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:23:41.128375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:41.128375+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:50.839701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"130'12714","reported_seq":12918,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.272172+0000","last_change":"2024-03-24T11:23:58.120751+0000","last_active":"2024-03-24T11:24:03.272172+0000","last_peered":"2024-03-24T11:24:03.272172+0000","last_clean":"2024-03-24T11:24:03.272172+0000","last_became_active":"2024-03-24T11:20:52.246358+0000","last_became_peered":"2024-03-24T11:20:52.246358+0000","last_unstale":"2024-03-24T11:24:03.272172+0000","last_undegraded":"2024-03-24T11:24:03.272172+0000","last_fullsized":"2024-03-24T11:24:03.272172+0000","mapping_epoch":92,"log_start":"110'5800","ondisk_log_start":"110'5800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'12386","last_scrub_stamp":"2024-03-24T11:23:58.120238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:58.120238+0000","objects_scrubbed":2734,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:13.845732+0000","scrub_duration":1230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156368896,"num_objects":2386,"num_object_clones":0,"num_object_copies":7158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12714,"num_write_kb":483200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.932960+0000","last_change":"2024-03-24T11:23:36.832426+0000","last_active":"2024-03-24T11:23:58.932960+0000","last_peered":"2024-03-24T11:23:58.932960+0000","last_clean":"2024-03-24T11:23:58.932960+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:23:58.932960+0000","last_undegraded":"2024-03-24T11:23:58.932960+0000","last_fullsized":"2024-03-24T11:23:58.932960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:36.832315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:36.832315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.760056+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.929817+0000","last_change":"2024-03-24T11:23:38.860651+0000","last_active":"2024-03-24T11:23:58.929817+0000","last_peered":"2024-03-24T11:23:58.929817+0000","last_clean":"2024-03-24T11:23:58.929817+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:23:58.929817+0000","last_undegraded":"2024-03-24T11:23:58.929817+0000","last_fullsized":"2024-03-24T11:23:58.929817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:38.860433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:38.860433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:00.422392+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":171,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.916707+0000","last_change":"2024-03-24T11:22:58.292666+0000","last_active":"2024-03-24T11:23:58.916707+0000","last_peered":"2024-03-24T11:23:58.916707+0000","last_clean":"2024-03-24T11:23:58.916707+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:23:58.916707+0000","last_undegraded":"2024-03-24T11:23:58.916707+0000","last_fullsized":"2024-03-24T11:23:58.916707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:22:58.292541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:22:58.292541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:10.214152+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"88'25","reported_seq":221,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.913697+0000","last_change":"2024-03-24T11:23:47.109191+0000","last_active":"2024-03-24T11:23:58.913697+0000","last_peered":"2024-03-24T11:23:58.913697+0000","last_clean":"2024-03-24T11:23:58.913697+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:23:58.913697+0000","last_undegraded":"2024-03-24T11:23:58.913697+0000","last_fullsized":"2024-03-24T11:23:58.913697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'25","last_scrub_stamp":"2024-03-24T11:23:47.109049+0000","last_deep_scrub":"88'25","last_deep_scrub_stamp":"2024-03-24T11:20:24.094499+0000","last_clean_scrub_stamp":"2024-03-24T11:23:47.109049+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:52.314842+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"130'12638","reported_seq":12844,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:24:03.663772+0000","last_change":"2024-03-24T11:23:37.402340+0000","last_active":"2024-03-24T11:24:03.663772+0000","last_peered":"2024-03-24T11:24:03.663772+0000","last_clean":"2024-03-24T11:24:03.663772+0000","last_became_active":"2024-03-24T11:20:52.240605+0000","last_became_peered":"2024-03-24T11:20:52.240605+0000","last_unstale":"2024-03-24T11:24:03.663772+0000","last_undegraded":"2024-03-24T11:24:03.663772+0000","last_fullsized":"2024-03-24T11:24:03.663772+0000","mapping_epoch":92,"log_start":"115'7000","ondisk_log_start":"115'7000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'11009","last_scrub_stamp":"2024-03-24T11:23:37.401802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:51.218936+0000","last_clean_scrub_stamp":"2024-03-24T11:23:37.401802+0000","objects_scrubbed":4002,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:47.776997+0000","scrub_duration":1536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152829952,"num_objects":2332,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12638,"num_write_kb":479040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.914197+0000","last_change":"2024-03-24T11:22:09.264563+0000","last_active":"2024-03-24T11:23:58.914197+0000","last_peered":"2024-03-24T11:23:58.914197+0000","last_clean":"2024-03-24T11:23:58.914197+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:23:58.914197+0000","last_undegraded":"2024-03-24T11:23:58.914197+0000","last_fullsized":"2024-03-24T11:23:58.914197+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:09.264454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:22:09.264454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":227,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.927796+0000","last_change":"2024-03-24T11:23:07.005268+0000","last_active":"2024-03-24T11:23:58.927796+0000","last_peered":"2024-03-24T11:23:58.927796+0000","last_clean":"2024-03-24T11:23:58.927796+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:23:58.927796+0000","last_undegraded":"2024-03-24T11:23:58.927796+0000","last_fullsized":"2024-03-24T11:23:58.927796+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:23:07.004669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:23:07.004669+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:08.010057+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":157,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.919358+0000","last_change":"2024-03-24T11:23:06.957259+0000","last_active":"2024-03-24T11:23:58.919358+0000","last_peered":"2024-03-24T11:23:58.919358+0000","last_clean":"2024-03-24T11:23:58.919358+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:23:58.919358+0000","last_undegraded":"2024-03-24T11:23:58.919358+0000","last_fullsized":"2024-03-24T11:23:58.919358+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:06.956906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:06.956906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:08.804192+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":168,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-24T11:23:58.917051+0000","last_change":"2024-03-24T11:22:59.172445+0000","last_active":"2024-03-24T11:23:58.917051+0000","last_peered":"2024-03-24T11:23:58.917051+0000","last_clean":"2024-03-24T11:23:58.917051+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:23:58.917051+0000","last_undegraded":"2024-03-24T11:23:58.917051+0000","last_fullsized":"2024-03-24T11:23:58.917051+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:22:59.172283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:22:59.172283+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:27.739692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2452422680,"num_objects":37422,"num_object_clones":0,"num_object_copies":112266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200962,"num_write_kb":7628225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7361867776,"data_stored":7361855560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90114,"ondisk_log_size":90114,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":303,"num_read_kb":298,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":440,"ondisk_log_size":440,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070530,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865532,"kb_used_data":766260,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92879620,"statfs":{"total":95995035648,"available":95108730880,"internally_reserved":0,"allocated":784650240,"data_stored":784333258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":101579211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364871,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862516,"kb_used_data":759148,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92882636,"statfs":{"total":95995035648,"available":95111819264,"internally_reserved":0,"allocated":777367552,"data_stored":777059007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022872,"kb_used_data":915280,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92722280,"statfs":{"total":95995035648,"available":94947614720,"internally_reserved":0,"allocated":937246720,"data_stored":936901604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1351176,"kb_used_data":1215872,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92393976,"statfs":{"total":95995035648,"available":94611431424,"internally_reserved":0,"allocated":1245052928,"data_stored":1244723437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1010484,"kb_used_data":907116,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92734668,"statfs":{"total":95995035648,"available":94960300032,"internally_reserved":0,"allocated":928886784,"data_stored":928576966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520848,"kb_used_data":1363336,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92224304,"statfs":{"total":95995035648,"available":94437687296,"internally_reserved":0,"allocated":1396056064,"data_stored":1395718308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986257,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":867012,"kb_used_data":763644,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92878140,"statfs":{"total":95995035648,"available":95107215360,"internally_reserved":0,"allocated":781971456,"data_stored":781646166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659212,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694192,"kb_used_data":622760,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93050960,"statfs":{"total":95995035648,"available":95284183040,"internally_reserved":0,"allocated":637706240,"data_stored":637400784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1218514944,"data_stored":1218510872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":928256000,"data_stored":928251928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":917110784,"data_stored":917110784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1373110272,"data_stored":1373110272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":761135104,"data_stored":761135104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":623181824,"data_stored":623181824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":772280320,"data_stored":772276248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":768278528,"data_stored":768278528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:24:09.625 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T11:24:09.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-24T11:24:09.666 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:09.689 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:09.753 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:09.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:09.767 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:09 smithi179 ceph-mon[29608]: pgmap v189: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 628 op/s 2024-03-24T11:24:09.768 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:09 smithi179 ceph-mon[37907]: pgmap v189: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 628 op/s 2024-03-24T11:24:09.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:09.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:09.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:09.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:09 smithi181 ceph-mon[33998]: pgmap v189: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 628 op/s 2024-03-24T11:24:09.944 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:09.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:10.062 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:10.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:10.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:10.196 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:10.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:10.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:10.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:10.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:10.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:10.487 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:10.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:10.589 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:10.628 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:10.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:10.729 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:10.773 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:10.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:10.874 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:10.919 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:10.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:11.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:11.063 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:11.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:11.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:11.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:11.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:11.299 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:11.338 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:11.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:11.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:11.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:11.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:11.531 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:24:11.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:11.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:11.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:11.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:11.756 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:11.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:11.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:11.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:11.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:11 smithi181 ceph-mon[33998]: pgmap v190: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-03-24T11:24:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:11 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2469633738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:11 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:11 smithi179 ceph-mon[29608]: pgmap v190: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-03-24T11:24:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2469633738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:11 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:11 smithi179 ceph-mon[37907]: pgmap v190: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 998 op/s 2024-03-24T11:24:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:11 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2469633738' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:11 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:11.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:12.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:12.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:12.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:12.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:12.303 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:12.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:12.440 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:12.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:12.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:12.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:12.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:12.800 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:12.829 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:12 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:12 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:24:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:12 smithi181 ceph-mon[33998]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:24:12.931 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:12 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:12 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:24:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:12 smithi179 ceph-mon[29608]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:24:12.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:12 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:12 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:24:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:12 smithi179 ceph-mon[37907]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T11:24:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:12.975 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:12.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:13.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:13.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:13.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:13.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:13.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:13.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.371 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:13.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:13.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:13.510 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:13.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:24:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:24:13.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:13.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:13 smithi179 ceph-mon[29608]: pgmap v192: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 642 op/s 2024-03-24T11:24:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:13 smithi179 ceph-mon[37907]: pgmap v192: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 642 op/s 2024-03-24T11:24:13.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:13.797 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:13.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:13.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:13 smithi181 ceph-mon[33998]: pgmap v192: 148 pgs: 148 active+clean; 2.0 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 642 op/s 2024-03-24T11:24:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:13.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:13.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:14.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:14.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:14.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:14.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:14.219 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:14.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:14.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:14.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:24:14.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:14.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:14.450 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:14.506 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:14.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:14.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:14.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:14.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:14 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:24:14.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:14 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:24:14.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:14.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:14 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:24:14.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:14 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:24:14.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:14.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:14.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:14.808 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:14.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:14 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:24:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:14 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:24:14.909 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:14.950 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:14.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:15.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:15.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:15.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:15.117 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:15.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:15.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:15.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:15.308 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:24:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:24:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:24:15.322 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:24:15.322 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:24:15.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:15.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:15.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:15.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:15.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:15.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:15.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:15 smithi179 ceph-mon[37907]: pgmap v193: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-24T11:24:15.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:15.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:15.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:15.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:15.775 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:15.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:15.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:15.893 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:15 smithi179 ceph-mon[29608]: pgmap v193: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-24T11:24:15.893 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:15 smithi179 ceph-mon[29608]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:24:15.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:15 smithi179 ceph-mon[37907]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:24:15.893 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:24:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:15 smithi181 ceph-mon[33998]: pgmap v193: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-24T11:24:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:15 smithi181 ceph-mon[33998]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T11:24:15.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:15.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:15.944 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:24:15.992 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:24:16.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:16.032 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:16.037 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:24:16.080 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:24:16.122 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:24:16.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:16.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:16.167 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:24:16.212 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:24:16.219 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:24:16.219 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:24:16.220 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:24:16.220 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:24:16.220 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:24:16.220 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:24:16.220 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:24:16.220 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:24:16.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.305 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:16.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:16.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:16.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:16.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.579 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:16.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:16.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:16.709 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:16.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:16.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:16.869 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:16 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/573238480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:16.869 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:16 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/573238480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:16.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:16 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/573238480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:16.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:16.971 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:16.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:17.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:17.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:17.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:17.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:17.247 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:17.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:17.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:17.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:17.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:17.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:17.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:17.529 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:24:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:17.538 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:17.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:17.631 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:17.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:17.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:17.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:17.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:17.733 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:17 smithi179 ceph-mon[29608]: pgmap v195: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:24:17.733 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:17 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:17.733 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:17 smithi179 ceph-mon[37907]: pgmap v195: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:24:17.733 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:17 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:17.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:17.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:17.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:17.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:17 smithi181 ceph-mon[33998]: pgmap v195: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:24:17.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:17 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:17.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:17.960 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:17.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:18.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:18.127 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:18.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:18.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:18.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:18.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:18.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:18.406 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:18.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:18.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:18.543 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:18.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:18.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:18.682 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:18.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:18.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:18.821 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:18.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:18.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:18.950 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:18.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:19.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:19.075 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:19.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:19.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:24:19.178 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:19.230 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:19.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:19.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:19.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:19.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:19.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:19.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:19.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:19.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:19.635 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:19.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:19.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:19.783 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:19.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:19.790 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:19 smithi179 ceph-mon[29608]: pgmap v196: 148 pgs: 148 active+clean; 1.6 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 752 op/s 2024-03-24T11:24:19.790 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:19 smithi179 ceph-mon[37907]: pgmap v196: 148 pgs: 148 active+clean; 1.6 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 752 op/s 2024-03-24T11:24:19.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:19.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:19 smithi181 ceph-mon[33998]: pgmap v196: 148 pgs: 148 active+clean; 1.6 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 752 op/s 2024-03-24T11:24:19.916 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:19.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:20.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:20.064 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:20.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:20.206 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:20.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:20.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:20.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:20.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:20.567 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:20.607 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:20.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:20.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:20.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.867 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:20.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:20.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:20 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:24:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:20 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:24:20.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:20 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:24:20.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:20 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:24:20.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:20 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:24:20.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:20 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:24:20.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:20.987 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:20.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:20.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:21.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:21.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:21.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:21.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:21.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:21.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:21.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:21.401 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:21.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:21.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:21.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:21.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:21.631 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 1 pg_num to 11 2024-03-24T11:24:21.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:21.680 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:21.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:21.782 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:21.824 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:21.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:21.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:21 smithi181 ceph-mon[33998]: pgmap v197: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:24:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:21 smithi181 ceph-mon[33998]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:24:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:21 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/968726326' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:24:21.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:21.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:21.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:21 smithi179 ceph-mon[29608]: pgmap v197: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:24:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:21 smithi179 ceph-mon[29608]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:24:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:21 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/968726326' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:24:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:21 smithi179 ceph-mon[37907]: pgmap v197: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:24:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:21 smithi179 ceph-mon[37907]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T11:24:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:21 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/968726326' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:24:22.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:22.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:22.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:22.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:22.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:22.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:22.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:22.369 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:22.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:22.380 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:22.470 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:22.522 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:22.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:22.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:22.635 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:24:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:22.656 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:22.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:22.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:22.801 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:22.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:22.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:22 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/968726326' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:24:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:22 smithi181 ceph-mon[33998]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:24:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:22 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:22.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:22 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:22.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:22 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2017882741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:24:22.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:22.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/968726326' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:24:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[29608]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:24:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2017882741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:24:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/968726326' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:24:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[37907]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T11:24:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:22 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2017882741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:24:23.037 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:23.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:23.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:23.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:23.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:23.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:23.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:23.384 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:23.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:23.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:23.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:23.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:23.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:23.623 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:24:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:24:23.661 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:23.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:23.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:23.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:23.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:23.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:23.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:23 smithi181 ceph-mon[33998]: pgmap v200: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 642 op/s 2024-03-24T11:24:23.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:23 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:23 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:24:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2017882741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:24:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:23 smithi181 ceph-mon[33998]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:24:23.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:23.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[29608]: pgmap v200: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 642 op/s 2024-03-24T11:24:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:24:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2017882741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:24:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[29608]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:24:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[37907]: pgmap v200: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 642 op/s 2024-03-24T11:24:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:24:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2017882741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:24:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:23 smithi179 ceph-mon[37907]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T11:24:24.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:24.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:24.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:24.159 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:24.199 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:24.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:24.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:24.345 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:24.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:24.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:24.481 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:24.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:24.583 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:24.620 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:24.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:24.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:24.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:24.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:24.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:24.855 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:24.893 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:24.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:24:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:24 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:24:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T11:24:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:24:24.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:24:24.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:24.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:24:24.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:24:24.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:24 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:24:24.995 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:25.036 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:25.138 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:25.176 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:25.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:25.307 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:25.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.441 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:25.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:25.443 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:24:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:24:25.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:25.571 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:25.673 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[29608]: pgmap v203: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:24:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:24:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:24:25.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:24:25.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[29608]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:24:25.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[37907]: pgmap v203: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:24:25.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:24:25.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:24:25.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:24:25.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:25 smithi179 ceph-mon[37907]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:24:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.713 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:25.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:25.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:25.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:25.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:25 smithi181 ceph-mon[33998]: pgmap v203: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:24:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:25 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:24:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:25 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:24:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:25 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:24:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:25 smithi181 ceph-mon[33998]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T11:24:25.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:25.987 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:25.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:26.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:26.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:26.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:26.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:26.260 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:26.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:26.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:26.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:24:26.404 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:26.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:26.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:26.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:26.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:26.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:26.665 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:26.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:26.767 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:26.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:26.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:26.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:26.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:26 smithi181 ceph-mon[33998]: 7.b scrub starts 2024-03-24T11:24:26.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:26 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:24:26.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:26 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:24:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:26 smithi181 ceph-mon[33998]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:24:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:26 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:24:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:26 smithi181 ceph-mon[33998]: 6.1f scrub ok 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[29608]: 7.b scrub starts 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[29608]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[29608]: 6.1f scrub ok 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[37907]: 7.b scrub starts 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:24:26.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:24:26.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[37907]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T11:24:26.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:24:26.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:26 smithi179 ceph-mon[37907]: 6.1f scrub ok 2024-03-24T11:24:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:26.962 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:26.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:27.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:27.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:27.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:27.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:27.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:27.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:27.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:27.395 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:27.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:27.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:27.536 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:27.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:27.551 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:24:27.551 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:24:27.551 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:24:27.551 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:24:27.551 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.551 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:142537 7.6 7:60057c3e:::benchmark_data_smithi179_78267_object23343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-24T11:24:27.551 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:00.920786+0000", 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 86.617490496000002, 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18071146900000001, 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 142537 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920786+0000", 2024-03-24T11:24:27.552 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920786+0000", 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920788+0000", 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.449e-06 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920792+0000", 2024-03-24T11:24:27.553 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9199999999999997e-06 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920802+0000", 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6800000000000005e-06 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920812+0000", 2024-03-24T11:24:27.554 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.064e-05 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.980033+0000", 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.059220361999999999 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.980125+0000", 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2127999999999996e-05 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.555 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.980256+0000", 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013109800000000001 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.985707+0000", 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054516340000000003 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.998268+0000", 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012560449 2024-03-24T11:24:27.556 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:01.101389+0000", 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.103121319 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:01.101439+0000", 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9969000000000002e-05 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.557 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:01.101498+0000", 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8820999999999998e-05 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:148311 7.6 7:64996475:::benchmark_data_smithi179_78267_object29117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:07.446179+0000", 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 80.092097674000001, 2024-03-24T11:24:27.558 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18068843000000001, 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 148311 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446179+0000", 2024-03-24T11:24:27.559 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446179+0000", 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446180+0000", 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.189e-06 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446184+0000", 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0199999999999996e-06 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446201+0000", 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6697e-05 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446210+0000", 2024-03-24T11:24:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4829999999999999e-06 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.456052+0000", 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098422119999999995 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.456161+0000", 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109466 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.456270+0000", 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108256 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.505129+0000", 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048859800000000002 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.505364+0000", 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000234148 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.563 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.626742+0000", 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.121378905 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.626799+0000", 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7040000000000003e-05 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.626868+0000", 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8213999999999994e-05 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.564 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:149855 7.6 7:6b469030:::benchmark_data_smithi179_78267_object30661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 78.168653449999994, 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20188611300000001, 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149855 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.565 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.566 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.53e-07 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369626+0000", 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3609999999999999e-06 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369632+0000", 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7640000000000002e-06 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.567 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369637+0000", 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7400000000000004e-06 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.408308+0000", 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038670910000000003 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.568 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.408440+0000", 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132261 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.408566+0000", 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126103 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.412115+0000", 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035489990000000002 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.569 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.420871+0000", 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0087561029999999995 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571373+0000", 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150501522 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.570 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571442+0000", 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9660999999999996e-05 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571510+0000", 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7335999999999994e-05 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:149867 7.6 7:6202a7d3:::benchmark_data_smithi179_78267_object30673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:24:27.571 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:09.387451+0000", 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 78.150826271, 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18884748100000001, 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149867 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.572 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387451+0000", 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387451+0000", 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387452+0000", 2024-03-24T11:24:27.573 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.065e-06 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387456+0000", 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4510000000000002e-06 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387472+0000", 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6110999999999999e-05 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.574 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387481+0000", 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8140000000000007e-06 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.420916+0000", 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033434868999999999 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.421066+0000", 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015033900000000001 2024-03-24T11:24:27.575 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.421223+0000", 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015693200000000001 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571551+0000", 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150328132 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.576 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571591+0000", 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9356000000000001e-05 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.576173+0000", 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045823180000000002 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.576227+0000", 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4076999999999997e-05 2024-03-24T11:24:27.577 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.576298+0000", 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1017000000000001e-05 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.578 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:150547 7.6 7:6af38a5f:::benchmark_data_smithi179_78267_object31353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:10.237125+0000", 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 77.301152346999999, 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16952882499999999, 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 150547 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.579 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237125+0000", 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237125+0000", 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.580 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237126+0000", 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1909999999999999e-06 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237130+0000", 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0899999999999998e-06 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.581 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.582 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.582 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237154+0000", 2024-03-24T11:24:27.582 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4261999999999999e-05 2024-03-24T11:24:27.582 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237162+0000", 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9589999999999995e-06 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.369979+0000", 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13281731999999999 2024-03-24T11:24:27.583 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.370117+0000", 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013744399999999999 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.370252+0000", 2024-03-24T11:24:27.584 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135425 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.380524+0000", 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010271481000000001 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.380570+0000", 2024-03-24T11:24:27.585 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6026000000000003e-05 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.406540+0000", 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025970156000000001 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.586 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.406589+0000", 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8982e-05 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.406653+0000", 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4488999999999999e-05 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:150732 7.6 7:635577ad:::benchmark_data_smithi179_78267_object31538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:24:27.587 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:10.477882+0000", 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 77.060394599000006, 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16796960499999999, 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 150732 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.588 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477882+0000", 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477882+0000", 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477893+0000", 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0869999999999999e-05 2024-03-24T11:24:27.589 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477898+0000", 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.169e-06 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477912+0000", 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3733e-05 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.590 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477921+0000", 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8370000000000003e-06 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.567027+0000", 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.089106481000000001 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.567151+0000", 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012334299999999999 2024-03-24T11:24:27.591 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.567266+0000", 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011532900000000001 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.575882+0000", 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086157690000000006 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.579826+0000", 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039438290000000003 2024-03-24T11:24:27.592 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.645746+0000", 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065919926000000004 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.645786+0000", 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0173999999999999e-05 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.593 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.645852+0000", 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6144999999999998e-05 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:151596 7.6 7:6b62085a:::benchmark_data_smithi179_78267_object32402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:11.452170+0000", 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 76.086106608999998, 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16698094199999999, 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.594 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 151596 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452170+0000", 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.595 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452170+0000", 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452172+0000", 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.421e-06 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452176+0000", 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7319999999999998e-06 2024-03-24T11:24:27.596 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452190+0000", 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3608000000000001e-05 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452199+0000", 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4149999999999999e-06 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.597 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.464029+0000", 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011830139 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.464150+0000", 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012094700000000001 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.464269+0000", 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119095 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.598 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.468119+0000", 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003850315 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.497652+0000", 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029532553 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.599 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.619031+0000", 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.121379203 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.619092+0000", 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1082999999999998e-05 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.619151+0000", 2024-03-24T11:24:27.600 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9431000000000001e-05 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152478 7.6 7:61d14fe2:::benchmark_data_smithi179_78267_object33284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.413643+0000", 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 75.124634275999995, 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17386502400000001, 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.601 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152478 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413643+0000", 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.602 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413643+0000", 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413644+0000", 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7859999999999999e-06 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413650+0000", 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5300000000000004e-06 2024-03-24T11:24:27.603 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413663+0000", 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2855e-05 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413671+0000", 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2539999999999992e-06 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.604 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.526208+0000", 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11253737 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.526316+0000", 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.605 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.526431+0000", 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114227 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532755+0000", 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063247149999999999 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.606 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.543894+0000", 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011138830000000001 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587234+0000", 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043339487000000003 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.607 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587398+0000", 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016432 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587508+0000", 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010965 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.608 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152480 7.6 7:6ab9905c:::benchmark_data_smithi179_78267_object33286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.417152+0000", 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 75.121124928, 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.170640984, 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152480 2024-03-24T11:24:27.609 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417152+0000", 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417152+0000", 2024-03-24T11:24:27.610 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417154+0000", 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0140000000000001e-06 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417160+0000", 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.02e-06 2024-03-24T11:24:27.611 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417182+0000", 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1991e-05 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417192+0000", 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0131e-05 2024-03-24T11:24:27.612 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.613 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.529357+0000", 2024-03-24T11:24:27.613 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.112164508 2024-03-24T11:24:27.613 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.613 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.613 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.529470+0000", 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112997 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.529588+0000", 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118303 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.614 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532801+0000", 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032128619999999999 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.549994+0000", 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017193109000000002 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587518+0000", 2024-03-24T11:24:27.615 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037523749000000002 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587732+0000", 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021424399999999999 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587793+0000", 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1056000000000006e-05 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.616 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152481 7.6 7:68b334a5:::benchmark_data_smithi179_78267_object33287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 75.121065395000002, 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.170816564, 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152481 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.617 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.618 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4600000000000005e-07 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417243+0000", 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0583999999999997e-05 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417252+0000", 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5100000000000004e-06 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.619 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417259+0000", 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5409999999999997e-06 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532833+0000", 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.115574467 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532950+0000", 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117262 2024-03-24T11:24:27.620 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.533064+0000", 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011379699999999999 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.543941+0000", 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010877006999999999 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.550033+0000", 2024-03-24T11:24:27.621 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006091539 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587801+0000", 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037767884000000002 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587965+0000", 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016419699999999999 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.622 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.588028+0000", 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3230000000000003e-05 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152618 7.6 7:6556be39:::benchmark_data_smithi179_78267_object33424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.603804+0000", 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 74.934472575000001, 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.175178431, 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.623 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152618 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603804+0000", 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603804+0000", 2024-03-24T11:24:27.624 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603805+0000", 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8299999999999996e-07 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603809+0000", 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8850000000000001e-06 2024-03-24T11:24:27.625 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603819+0000", 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7259999999999997e-06 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603826+0000", 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1879999999999996e-06 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.626 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.717660+0000", 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.113833881 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.717774+0000", 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114626 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.717894+0000", 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119498 2024-03-24T11:24:27.627 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.724573+0000", 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066788280000000004 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.730616+0000", 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006043484 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.628 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.778702+0000", 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048085485999999997 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.778925+0000", 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022367599999999999 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.778983+0000", 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7469999999999997e-05 2024-03-24T11:24:27.629 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.630 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.630 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.630 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.630 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.630 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153073 7.6 7:65c05f1d:::benchmark_data_smithi179_78267_object33879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.630 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.201863+0000", 2024-03-24T11:24:27.631 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 74.336414309000006, 2024-03-24T11:24:27.631 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.184056108, 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153073 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201863+0000", 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.632 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201863+0000", 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201864+0000", 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.522e-06 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.633 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201870+0000", 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8810000000000001e-06 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201891+0000", 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0806999999999999e-05 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201899+0000", 2024-03-24T11:24:27.634 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0160000000000008e-06 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.249353+0000", 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.047454601999999999 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.635 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.249517+0000", 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016394399999999999 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.249652+0000", 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013499200000000001 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.253703+0000", 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040510349999999997 2024-03-24T11:24:27.636 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.256442+0000", 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027389210000000001 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.385800+0000", 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.129357953 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.637 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.385855+0000", 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4255000000000001e-05 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.385919+0000", 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4179999999999999e-05 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.638 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153169 7.6 7:6261c8e3:::benchmark_data_smithi179_78267_object33975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.362517+0000", 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 74.175759604999996, 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20489932799999999, 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153169 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.639 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362517+0000", 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362517+0000", 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.640 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362519+0000", 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5910000000000001e-06 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362523+0000", 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0010000000000003e-06 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362536+0000", 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3351e-05 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.641 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362545+0000", 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0270000000000001e-06 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.404648+0000", 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.042102939999999998 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.642 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.404769+0000", 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120915 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.404929+0000", 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000160056 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.643 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.410256+0000", 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053271070000000002 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.420180+0000", 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099239410000000004 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.644 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.567303+0000", 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14712319099999999 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.567361+0000", 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7936999999999998e-05 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.645 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.567417+0000", 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5271e-05 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153604 7.6 7:6abf53f5:::benchmark_data_smithi179_78267_object34410:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.918161+0000", 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.620115534000007, 2024-03-24T11:24:27.646 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18721666100000001, 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153604 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.647 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918161+0000", 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918161+0000", 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.648 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918164+0000", 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4090000000000001e-06 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918170+0000", 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0870000000000003e-06 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918184+0000", 2024-03-24T11:24:27.649 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4406999999999999e-05 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918198+0000", 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3628999999999999e-05 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939774+0000", 2024-03-24T11:24:27.650 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021575596999999998 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939928+0000", 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015411799999999999 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.940048+0000", 2024-03-24T11:24:27.651 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120003 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.952963+0000", 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012915714999999999 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.652 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105236+0000", 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.152272716 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105256+0000", 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9993e-05 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.653 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105314+0000", 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7762e-05 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105378+0000", 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4225000000000003e-05 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.654 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153605 7.6 7:6583e64f:::benchmark_data_smithi179_78267_object34411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.918217+0000", 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.620059639000004, 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.187972839, 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153605 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.655 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918217+0000", 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918217+0000", 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918218+0000", 2024-03-24T11:24:27.656 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0800000000000005e-07 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918221+0000", 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0010000000000002e-06 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918238+0000", 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7331000000000001e-05 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.657 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918248+0000", 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9969999999999993e-06 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.943141+0000", 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024892986999999998 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.943227+0000", 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6169999999999997e-05 2024-03-24T11:24:27.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.943299+0000", 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1586000000000002e-05 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953002+0000", 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009702769 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105389+0000", 2024-03-24T11:24:27.659 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15238744000000001 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105958+0000", 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056881200000000003 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106010+0000", 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2571000000000002e-05 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.660 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106190+0000", 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000179667 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153610 7.6 7:6b61266a:::benchmark_data_smithi179_78267_object34416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.924956+0000", 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.613320536000003, 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.181386716, 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153610 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924956+0000", 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.662 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924956+0000", 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924958+0000", 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4759999999999999e-06 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.663 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924963+0000", 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8749999999999999e-06 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924983+0000", 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0017000000000002e-05 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.664 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924992+0000", 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7479999999999999e-06 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.946269+0000", 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021276994 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.946408+0000", 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013992099999999999 2024-03-24T11:24:27.665 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.946534+0000", 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012597899999999999 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953037+0000", 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065029839999999998 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.666 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105396+0000", 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15235836999999999 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106249+0000", 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00085298399999999999 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106296+0000", 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7108999999999999e-05 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.667 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106343+0000", 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7259000000000003e-05 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153614 7.6 7:67b9b693:::benchmark_data_smithi179_78267_object34420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.929484+0000", 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.608793274999996, 2024-03-24T11:24:27.668 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17737636900000001, 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153614 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929484+0000", 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.669 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929484+0000", 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929485+0000", 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.381e-06 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.670 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929492+0000", 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7750000000000004e-06 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929509+0000", 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6708e-05 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929518+0000", 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2790000000000004e-06 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.671 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.949499+0000", 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019981298000000002 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.949635+0000", 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135947 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.949764+0000", 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012860500000000001 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.672 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953076+0000", 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003312551 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105403+0000", 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15232652599999999 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106432+0000", 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001029248 2024-03-24T11:24:27.673 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106661+0000", 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022862800000000001 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106860+0000", 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000199423 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.674 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:153608 7.e e121/92 7:7b7381e4:::benchmark_data_smithi179_78267_object34414:head v 121'9772, mlcod=121'9767)", 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.934510+0000", 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.603767314999999, 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17072061699999999, 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934510+0000", 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.675 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934510+0000", 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934511+0000", 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.071e-06 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934518+0000", 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6179999999999997e-06 2024-03-24T11:24:27.676 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934533+0000", 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4301000000000001e-05 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934541+0000", 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1459999999999996e-06 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939257+0000", 2024-03-24T11:24:27.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004716273 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939277+0000", 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9683000000000001e-05 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105193+0000", 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16591620500000001 2024-03-24T11:24:27.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105230+0000", 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7320000000000002e-05 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153615 7.6 7:61f98c16:::benchmark_data_smithi179_78267_object34421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.936023+0000", 2024-03-24T11:24:27.679 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.602254213999998, 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.1709802, 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153615 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936023+0000", 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936023+0000", 2024-03-24T11:24:27.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936024+0000", 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3689999999999999e-06 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936030+0000", 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.524e-06 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936047+0000", 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6991999999999999e-05 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936055+0000", 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6130000000000004e-06 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953105+0000", 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017049983000000001 2024-03-24T11:24:27.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953221+0000", 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011586600000000001 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953346+0000", 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012509199999999999 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105128+0000", 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15178178000000001 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105410+0000", 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028236199999999998 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106906+0000", 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014960349999999999 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106956+0000", 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0071999999999998e-05 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.107003+0000", 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6511999999999998e-05 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153967 7.6 7:6c5dc8d3:::benchmark_data_smithi179_78267_object34773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:24:27.687 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:14.442129+0000", 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 73.096147521000006, 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18404717000000001, 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153967 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442129+0000", 2024-03-24T11:24:27.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442129+0000", 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442131+0000", 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3540000000000001e-06 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:24:27.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442133+0000", 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6709999999999999e-06 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442141+0000", 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.644e-06 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:24:27.690 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442147+0000", 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5890000000000002e-06 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.607217+0000", 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16507073999999999 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.607383+0000", 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016527600000000001 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.607505+0000", 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012231000000000001 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.613531+0000", 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060264079999999996 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.625923+0000", 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012391154 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.626059+0000", 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013637600000000001 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.626114+0000", 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5143999999999999e-05 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.626177+0000", 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2503999999999997e-05 2024-03-24T11:24:27.693 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.694 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.694 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.694 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:24:27.694 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:24:27.694 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:24:27.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:27.734 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:27.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:27.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:27.877 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:27.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:27.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:27 smithi181 ceph-mon[33998]: pgmap v206: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:24:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:27 smithi181 ceph-mon[33998]: 7.b scrub ok 2024-03-24T11:24:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:27 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:24:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:27 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:24:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:27 smithi181 ceph-mon[33998]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:24:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[29608]: pgmap v206: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:24:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[29608]: 7.b scrub ok 2024-03-24T11:24:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:24:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:24:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[29608]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:24:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[37907]: pgmap v206: 158 pgs: 1 peering, 10 unknown, 147 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:24:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[37907]: 7.b scrub ok 2024-03-24T11:24:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:24:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:24:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:27 smithi179 ceph-mon[37907]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T11:24:27.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:28.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:28.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.157 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:28.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:28.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.302 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:28.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:28.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:28.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:28.476 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:28.507 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:24:28.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:28.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:28.571 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:24:28.629 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:24:28.660 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:24:28.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:28.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:28.698 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:24:28.735 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:24:28.776 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:24:28.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:28.813 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:24:28.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:28.825 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:24:28.825 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:24:28.826 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:24:28.826 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:24:28.826 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:24:28.826 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:24:28.826 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:24:28.826 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:24:28.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:28.946 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:28.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:29.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:29.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:29.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:29.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:29.216 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:29.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:29.317 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:29.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:29.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:29.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:24:29.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:24:29.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:24:29.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:24:29.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: pgmap v209: 158 pgs: 2 active+undersized+remapped, 8 unknown, 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 242 op/s 2024-03-24T11:24:29.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:24:29.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2450603077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:29.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:29 smithi181 ceph-mon[33998]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:24:29.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:29.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:24:29.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:24:29.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:24:29.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:24:29.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: pgmap v209: 158 pgs: 2 active+undersized+remapped, 8 unknown, 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 242 op/s 2024-03-24T11:24:29.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2450603077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[29608]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: pgmap v209: 158 pgs: 2 active+undersized+remapped, 8 unknown, 148 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 242 op/s 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2450603077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:29.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:29 smithi179 ceph-mon[37907]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T11:24:29.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:29.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:29.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:29.627 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:29.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:29.729 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:29.768 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:29.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:29.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:29.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:29.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:29.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:30.027 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:30.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:30.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:30.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.298 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:30.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:30.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 4.11 scrub starts 2024-03-24T11:24:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 4.11 scrub ok 2024-03-24T11:24:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:24:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:24:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:24:30.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:24:30.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:24:30.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:24:30.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:24:30.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:30 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:24:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:30.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:30.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 4.11 scrub starts 2024-03-24T11:24:30.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 4.11 scrub ok 2024-03-24T11:24:30.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:24:30.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 4.11 scrub starts 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 4.11 scrub ok 2024-03-24T11:24:30.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:24:30.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:24:30.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:24:30.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:24:30.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:24:30.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:24:30.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:24:30.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:30 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:24:30.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.554 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:30.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:30.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.698 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:30.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:30.799 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.823 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:30.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:30.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:30.954 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:30.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:31.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:31.098 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:31.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:31.199 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:31.238 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:31.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:31.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:31.360 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:24:31.360 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:24:31.361 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-24T11:24:31.361 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:24:31.361 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:24:31.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-24T11:24:31.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:31.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:31.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: pgmap v211: 158 pgs: 4 peering, 2 active+undersized+remapped, 4 unknown, 148 active+clean; 766 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s; 40 KiB/s, 0 objects/s recovering 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.4 scrub starts 2024-03-24T11:24:31.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:24:31.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.4 scrub ok 2024-03-24T11:24:31.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.2 scrub starts 2024-03-24T11:24:31.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 1.2 scrub ok 2024-03-24T11:24:31.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:24:31.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:31 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:24:31.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:24:31.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:24:31.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:24:31.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:24:31.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:24:31.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: pgmap v211: 158 pgs: 4 peering, 2 active+undersized+remapped, 4 unknown, 148 active+clean; 766 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s; 40 KiB/s, 0 objects/s recovering 2024-03-24T11:24:31.404 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.4 scrub starts 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.4 scrub ok 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.2 scrub starts 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 1.2 scrub ok 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:24:31.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: pgmap v211: 158 pgs: 4 peering, 2 active+undersized+remapped, 4 unknown, 148 active+clean; 766 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s; 40 KiB/s, 0 objects/s recovering 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.4 scrub starts 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.4 scrub ok 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.2 scrub starts 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 1.2 scrub ok 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:24:31.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:31 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:24:31.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:31.544 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:31.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:31.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:31.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:31.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:31.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:31.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:31.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:31.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:24:31.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:31.954 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:31.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:32.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:32.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:32.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:32.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:32.239 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:32.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:32.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:32.360 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:24:32.360 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:24:32.360 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:24:32.360 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:24:32.361 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:24:32.361 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:24:32.361 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:24:32.361 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:32 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:24:32.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:32.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:32.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:32 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:24:32.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:32 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:24:32.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:32 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:24:32.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:32 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:24:32.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:32.519 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:32.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:32.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:32.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:32.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:32.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:32.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:32.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:32.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:32.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:33.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:33.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:33.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 1.8 scrub starts 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 1.8 scrub ok 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 7.2 scrub starts 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: pgmap v212: 158 pgs: 4 peering, 2 active+undersized+remapped, 4 unknown, 148 active+clean; 766 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 27 KiB/s, 0 objects/s recovering 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 7.2 scrub ok 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:24:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:24:33.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:24:33.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:33.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:33.219 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 1.8 scrub starts 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 1.8 scrub ok 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 7.2 scrub starts 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: pgmap v212: 158 pgs: 4 peering, 2 active+undersized+remapped, 4 unknown, 148 active+clean; 766 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 27 KiB/s, 0 objects/s recovering 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 7.2 scrub ok 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:24:33.220 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 1.8 scrub starts 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 1.8 scrub ok 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 7.2 scrub starts 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: pgmap v212: 158 pgs: 4 peering, 2 active+undersized+remapped, 4 unknown, 148 active+clean; 766 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 27 KiB/s, 0 objects/s recovering 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 7.2 scrub ok 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:24:33.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:33.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:33.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:33.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:33.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:33.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:33.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:33.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:33.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:33.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:33.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:33.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:33.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:24:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:24:33.750 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:33.806 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:33.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:33.863 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:33.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:33.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:33.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:33.947 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:34.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:34.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:34.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:34.125 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:24:34.126 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[29608]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:24:34.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:34.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:34.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:34.293 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:34.344 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:34.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:24:34.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:34 smithi181 ceph-mon[33998]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:24:34.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:24:34.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:34 smithi179 ceph-mon[37907]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T11:24:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:34.512 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:34.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:34.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:34.641 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:34.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:34.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:34.768 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:34.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:34.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:34.901 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:34.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:35.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:35.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:35.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:35.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:35.173 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:35.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:35.276 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:35.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:35.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: pgmap v214: 158 pgs: 158 active+clean; 451 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:24:35.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:35 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:24:35.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:35.453 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:35.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:35.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:24:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:24:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:24:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:24:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: pgmap v214: 158 pgs: 158 active+clean; 451 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:24:35.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: pgmap v214: 158 pgs: 158 active+clean; 451 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.51k op/s 2024-03-24T11:24:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:24:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:24:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:24:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:35 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:24:35.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:35.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:35.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:35.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:35.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:35.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:35.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:35.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:35.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:35.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:24:35.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.000 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:36.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:36.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:36.151 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:24:36.157 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:36.259 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:36.303 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:36.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:36 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:24:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:36 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:24:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2972205538' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:36 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:36 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:24:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:36 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:24:36.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:36.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:24:36.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:24:36.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2972205538' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:36.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:36.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:24:36.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:24:36.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:24:36.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:24:36.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2972205538' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:36.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:24:36.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:24:36.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:36 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:24:36.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:36.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:36.573 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:36.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:36.703 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:36.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:36.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:36.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:36.969 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:36.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:37.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:37.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:37.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:37.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:37.239 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:37.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:37.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:37.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:37.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:37.397 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:24:37.397 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:24:37.397 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:24:37.397 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:24:37.398 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: pgmap v215: 158 pgs: 158 active+clean; 451 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-03-24T11:24:37.398 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:24:37.398 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:24:37.398 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:24:37.398 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[29608]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:24:37.398 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: pgmap v215: 158 pgs: 158 active+clean; 451 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:24:37.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:37 smithi179 ceph-mon[37907]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:24:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:24:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:24:37.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:24:37.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:24:37.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: pgmap v215: 158 pgs: 158 active+clean; 451 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-03-24T11:24:37.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:24:37.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:24:37.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:24:37.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:37 smithi181 ceph-mon[33998]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T11:24:37.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:37.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:37.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:37.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:37.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:37.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:37.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:37.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:37.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:37.764 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:24:37.764 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T11:24:37.764 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:24:37.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:37.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:37.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:37.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:38.063 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:38.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:38.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:38.194 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:38.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:38.295 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:38.337 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:38.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:38.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:38.445 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:38 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:24:38.445 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:38 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:24:38.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:38 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:24:38.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:38 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:24:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:38.478 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:38.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:38.579 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:38.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:38.624 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:38.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:38.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:38 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:24:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:38 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:24:38.725 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:38.771 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:38.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:38.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:38.917 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:38.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:39.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:39.058 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:39.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:39.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:39.202 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:39.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[29608]: pgmap v217: 158 pgs: 158 active+clean; 392 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[29608]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[37907]: pgmap v217: 158 pgs: 158 active+clean; 392 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-03-24T11:24:39.253 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:39 smithi179 ceph-mon[37907]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:24:39.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:39.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:39.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:39.437 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:39.450 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:24:39.487 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:39.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:39.527 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:39.589 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:39.627 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:24:39.628 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:24:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:39.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:39.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:39.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:39 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:24:39.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:39 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:24:39.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:39 smithi181 ceph-mon[33998]: pgmap v217: 158 pgs: 158 active+clean; 392 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-03-24T11:24:39.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:39 smithi181 ceph-mon[33998]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T11:24:39.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:39.812 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:39.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:39.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:39.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:39.962 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:39.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:40.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:40.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:40.185 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Removed 119191 objects 2024-03-24T11:24:40.186 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Clean up completed and total clean up time :123.093 2024-03-24T11:24:40.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:40.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:40.317 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.341 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:40.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:40.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[29608]: 6.e scrub starts 2024-03-24T11:24:40.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[29608]: 6.e scrub ok 2024-03-24T11:24:40.405 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[29608]: 7.a scrub starts 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[29608]: 7.a scrub ok 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[37907]: 6.e scrub starts 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[37907]: 6.e scrub ok 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[37907]: 7.a scrub starts 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[37907]: 7.a scrub ok 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:24:40.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:40 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:24:40.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.467 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:40.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:40.541 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-24T11:24:40.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-24T11:24:40.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:40.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:40.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:40 smithi181 ceph-mon[33998]: 6.e scrub starts 2024-03-24T11:24:40.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:40 smithi181 ceph-mon[33998]: 6.e scrub ok 2024-03-24T11:24:40.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:40 smithi181 ceph-mon[33998]: 7.a scrub starts 2024-03-24T11:24:40.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:40 smithi181 ceph-mon[33998]: 7.a scrub ok 2024-03-24T11:24:40.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:40 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:24:40.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:40 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:24:40.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:40.719 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:40.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:40.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:40.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:40.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:40.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:41.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:41.098 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:41.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:41.199 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:41.226 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:41.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:41.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:41.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:41.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:41.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:41.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:41 smithi179 ceph-mon[29608]: pgmap v219: 158 pgs: 158 active+clean; 119 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 846 B/s rd, 0 B/s wr, 878 op/s 2024-03-24T11:24:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:41 smithi179 ceph-mon[37907]: pgmap v219: 158 pgs: 158 active+clean; 119 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 846 B/s rd, 0 B/s wr, 878 op/s 2024-03-24T11:24:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:41.482 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:41.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:41.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:24:41.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:41.620 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:41.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:41.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:41 smithi181 ceph-mon[33998]: pgmap v219: 158 pgs: 158 active+clean; 119 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 846 B/s rd, 0 B/s wr, 878 op/s 2024-03-24T11:24:41.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:41.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:41.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:41.851 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:41.876 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:41.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:41.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.002 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:42.103 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.128 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:42.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.253 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:42.273 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:42 smithi179 ceph-mon[29608]: 3.12 deep-scrub starts 2024-03-24T11:24:42.274 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:42 smithi179 ceph-mon[29608]: 3.12 deep-scrub ok 2024-03-24T11:24:42.274 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:42 smithi179 ceph-mon[37907]: 3.12 deep-scrub starts 2024-03-24T11:24:42.274 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:42 smithi179 ceph-mon[37907]: 3.12 deep-scrub ok 2024-03-24T11:24:42.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:42.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:42.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.507 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:42.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:42.634 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:42 smithi181 ceph-mon[33998]: 3.12 deep-scrub starts 2024-03-24T11:24:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:42 smithi181 ceph-mon[33998]: 3.12 deep-scrub ok 2024-03-24T11:24:42.717 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:42.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:42.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:42.884 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:42.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:42.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.012 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:43.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:43.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:43.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:43.235 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:43 smithi179 ceph-mon[29608]: pgmap v220: 158 pgs: 158 active+clean; 119 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 664 op/s 2024-03-24T11:24:43.235 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:43 smithi179 ceph-mon[37907]: pgmap v220: 158 pgs: 158 active+clean; 119 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 664 op/s 2024-03-24T11:24:43.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:43.242 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:43.266 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:43.285 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:43.290 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:24:43.315 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:43 smithi181 ceph-mon[33998]: pgmap v220: 158 pgs: 158 active+clean; 119 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 664 op/s 2024-03-24T11:24:43.316 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:24:43.335 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:24:43.354 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:24:43.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:43.374 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:24:43.394 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:24:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:43.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:43.414 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:24:43.435 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:24:43.436 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:24:43.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.523 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:43.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:43.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:43.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:24:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:24:43.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:43.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.780 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:43.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:43.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:43.906 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:43.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:44.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:44.024 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:24:44.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:24:44.025 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-24T11:24:44.025 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-24T11:24:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.031 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:44.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:44.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:44.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:44.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:44.285 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:44.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.416 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:44.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:44.442 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: 5.f deep-scrub starts 2024-03-24T11:24:44.442 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: 5.f deep-scrub ok 2024-03-24T11:24:44.442 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:24:44.442 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:24:44.442 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3176728470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:44.442 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: 5.f deep-scrub starts 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: 5.f deep-scrub ok 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3176728470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:24:44.443 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:44 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:24:44.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.541 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:44.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:44.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: 5.f deep-scrub starts 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: 5.f deep-scrub ok 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3176728470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:24:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:44 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:24:44.667 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:44.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:44.796 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:44.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:44.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:44.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:45.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:45.052 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:45.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:45.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:45.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.307 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:45.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:45.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:45.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:45.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:24:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:24:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:24:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:24:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:24:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: pgmap v221: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: pgmap v221: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:24:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:24:45.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:24:45.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:45 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:24:45.538 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.562 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:45.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: pgmap v221: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 901 op/s 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:24:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:45 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:24:45.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:45.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:45.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:45.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:45.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:45.944 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:45.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:46.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.069 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:46.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:46.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:46.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:46.233 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:24:46.233 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:24:46.233 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:24:46.233 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:24:46.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:24:46.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:24:46.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:24:46.234 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:46 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:24:46.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.322 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:46.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:46.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:46.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:46.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.579 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:46.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:46 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:24:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:46 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:24:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:46 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:24:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:46 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:24:46.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:46.705 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:46.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:46.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:46.937 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:46.963 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:46.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:47.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:47.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:47.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:47.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:47.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:47.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:47.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:47.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:47.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:47.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[29608]: pgmap v222: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 722 op/s 2024-03-24T11:24:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:24:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:24:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[29608]: 3.f deep-scrub starts 2024-03-24T11:24:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[29608]: 3.f deep-scrub ok 2024-03-24T11:24:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[37907]: pgmap v222: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 722 op/s 2024-03-24T11:24:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:24:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:24:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[37907]: 3.f deep-scrub starts 2024-03-24T11:24:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:47 smithi179 ceph-mon[37907]: 3.f deep-scrub ok 2024-03-24T11:24:47.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:47.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:47.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:47.611 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:47.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:47.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:47 smithi181 ceph-mon[33998]: pgmap v222: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 722 op/s 2024-03-24T11:24:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:47 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:24:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:47 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:24:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:47 smithi181 ceph-mon[33998]: 3.f deep-scrub starts 2024-03-24T11:24:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:47 smithi181 ceph-mon[33998]: 3.f deep-scrub ok 2024-03-24T11:24:47.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:47.742 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:47.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:47.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:47.871 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:47.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:47.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:48.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:48.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:48.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:48.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:48.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:48.232 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:48.260 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:48.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:48.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:48.390 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:48.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:24:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:24:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:24:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:24:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:48.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:48.516 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:48.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:48.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:48.791 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:48 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:24:48.791 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:48 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:24:48.791 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:24:48.791 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:24:48.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:48.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:48.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:48.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:48.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:49.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:49.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:49.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:49.203 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:49.229 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:49.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:49.334 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:49.436 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:49.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[29608]: 7.0 scrub starts 2024-03-24T11:24:49.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[29608]: 7.0 scrub ok 2024-03-24T11:24:49.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[29608]: pgmap v223: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 626 op/s 2024-03-24T11:24:49.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[29608]: 6.1c scrub starts 2024-03-24T11:24:49.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[29608]: 6.1c scrub ok 2024-03-24T11:24:49.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/233606667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:24:49.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:24:49.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[37907]: 7.0 scrub starts 2024-03-24T11:24:49.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[37907]: 7.0 scrub ok 2024-03-24T11:24:49.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[37907]: pgmap v223: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 626 op/s 2024-03-24T11:24:49.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[37907]: 6.1c scrub starts 2024-03-24T11:24:49.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[37907]: 6.1c scrub ok 2024-03-24T11:24:49.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/233606667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:24:49.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:49 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:24:49.463 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:49.552 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:24:49.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:49.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:49 smithi181 ceph-mon[33998]: 7.0 scrub starts 2024-03-24T11:24:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:49 smithi181 ceph-mon[33998]: 7.0 scrub ok 2024-03-24T11:24:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:49 smithi181 ceph-mon[33998]: pgmap v223: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 626 op/s 2024-03-24T11:24:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:49 smithi181 ceph-mon[33998]: 6.1c scrub starts 2024-03-24T11:24:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:49 smithi181 ceph-mon[33998]: 6.1c scrub ok 2024-03-24T11:24:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:49 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/233606667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:24:49.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:49 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:24:49.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.716 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:49.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:49.777 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:24:49.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.843 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:49.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:49.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:49.969 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:49.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:50.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.096 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:50.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:50.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.221 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:50.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:50.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:50.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:50.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:50.457 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:24:50.457 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:24:50.457 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:24:50.457 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[29608]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:24:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:24:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:24:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:24:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:24:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:24:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:24:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:50 smithi179 ceph-mon[37907]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:24:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:50.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:50.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:50.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:24:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:24:50.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:50 smithi181 ceph-mon[33998]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T11:24:50.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:50.714 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:24:50.714 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:24:50.714 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:24:50.714 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:24:50.714 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:24:50.737 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:50.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:50.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:50.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:50.987 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:50.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:51.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:51.113 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:51.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:51.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:51.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:51.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:51.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:51.371 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:51.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:24:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[29608]: pgmap v225: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 189 op/s 2024-03-24T11:24:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:24:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3598097463' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:24:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:24:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:24:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:24:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:24:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[37907]: pgmap v225: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 189 op/s 2024-03-24T11:24:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:24:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3598097463' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:24:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:24:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:24:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:51 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:24:51.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:51.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:51.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:51.578 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_0' removed 2024-03-24T11:24:51.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:51.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:51.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:51 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:24:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:51 smithi181 ceph-mon[33998]: pgmap v225: 158 pgs: 158 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 189 op/s 2024-03-24T11:24:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:51 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:24:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3598097463' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:24:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:51 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:24:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:51 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:24:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:51 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:24:51.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:51.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:51.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:51.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:51.893 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:51.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:51.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:52.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:52.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:52.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:52.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:52.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:52.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.398 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:52.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:52.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.523 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:52.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:52.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:52.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:52.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:52.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:52.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:52.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:52.901 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:52 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:24:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:52 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:24:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:52 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:24:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:52 smithi181 ceph-mon[33998]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:24:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:52.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:52.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:24:52.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:24:52.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:24:52.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[29608]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:24:52.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:24:52.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:24:52.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:24:52.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:52 smithi179 ceph-mon[37907]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T11:24:53.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.028 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:53.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:53.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:53.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:53.223 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:24:53.257 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:53.291 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:53.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.417 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:53.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:53.515 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:24:53.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.582 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:53.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:53.616 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:53 smithi179 ceph-mon[29608]: pgmap v227: 142 pgs: 142 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:24:53.617 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:53 smithi179 ceph-mon[37907]: pgmap v227: 142 pgs: 142 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:24:53.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:53 smithi181 ceph-mon[33998]: pgmap v227: 142 pgs: 142 active+clean; 582 KiB data, 977 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:24:53.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:24:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:24:53.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:53.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:53.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:53.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:53.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:53.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:53.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:54.015 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:24:54.015 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:24:54.015 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:24:54.015 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:24:54.016 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:24:54.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:54.110 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:54.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:54.211 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:54.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:54.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:54.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:54.362 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:54.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:54.448 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T11:24:54.449 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T11:24:54.449 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-24T11:24:54.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-24T11:24:54.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:54.518 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:54.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:54.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:54.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:54.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:54.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:54.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:54.774 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:54.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:54.775 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:54.778 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:54.779 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:54.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:24:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:54.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:54.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:55.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:55.036 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:55.137 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:55.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:55.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:24:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:24:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:24:55.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:55.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:55.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:55.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:55.469 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:24:55.484 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:24:55.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-24T11:24:55.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:55.551 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:55.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.678 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:55.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:55 smithi179 ceph-mon[29608]: pgmap v228: 142 pgs: 142 active+clean; 582 KiB data, 998 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:24:55.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:55 smithi179 ceph-mon[37907]: pgmap v228: 142 pgs: 142 active+clean; 582 KiB data, 998 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:24:55.780 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.804 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:55.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:55 smithi181 ceph-mon[33998]: pgmap v228: 142 pgs: 142 active+clean; 582 KiB data, 998 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:24:55.905 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:55.929 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:55.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:56.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:56.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:56.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:56.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:56.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.316 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:56.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:56.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.446 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:56.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:56.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.574 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:56.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:56.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.701 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:56.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:56.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:56.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:56.854 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:56 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:24:56.854 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:56 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:24:56.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:56 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:24:56.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:56 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:24:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:56 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:24:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:56 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:24:56.932 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:56.957 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:56.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:57.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:57.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:57.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.213 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:57.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:57.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.342 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:57.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:57.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.468 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:57.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:57.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.594 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:57.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:57.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:57 smithi179 ceph-mon[29608]: pgmap v229: 142 pgs: 142 active+clean; 582 KiB data, 998 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:24:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:57 smithi179 ceph-mon[29608]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:24:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:57 smithi179 ceph-mon[37907]: pgmap v229: 142 pgs: 142 active+clean; 582 KiB data, 998 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:24:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:57 smithi179 ceph-mon[37907]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:24:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:57.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:57.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:57.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:57.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:57 smithi181 ceph-mon[33998]: pgmap v229: 142 pgs: 142 active+clean; 582 KiB data, 998 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:24:57.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:57 smithi181 ceph-mon[33998]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T11:24:57.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:57.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:57.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:58.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:58.114 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:58.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:58.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:58.241 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:58.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:58.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:24:58.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:58.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:24:58.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:24:58.499 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:24:58.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:58.630 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:58.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:58.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:58.757 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:58.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:58.806 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:24:58.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:58.883 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:58.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:58.984 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.009 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:59.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:59.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:24:59.133 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:24:59.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.261 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:59.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:59.362 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:24:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:59.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:24:59.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:24:59.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:24:59.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:24:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:59.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:24:59.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:24:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:24:59.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:24:59.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:59 smithi179 ceph-mon[29608]: pgmap v231: 142 pgs: 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:24:59.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:24:59 smithi179 ceph-mon[29608]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:24:59.860 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:59 smithi179 ceph-mon[37907]: pgmap v231: 142 pgs: 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:24:59.860 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:24:59 smithi179 ceph-mon[37907]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:24:59.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:24:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:24:59.899 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:24:59.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:24:59.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:59 smithi181 ceph-mon[33998]: pgmap v231: 142 pgs: 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:24:59.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:24:59 smithi181 ceph-mon[33998]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T11:25:00.000 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.026 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:00.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:00.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:00.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:00.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:00.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:00.385 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.410 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:00.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:00.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:00.536 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:00.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:00.662 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:00.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.789 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:00.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:00.832 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:25:00.832 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:25:00.832 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:25:00.833 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:25:00.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.833 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.14156.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-24T11:25:00.833 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:16:41.065262+0000", 2024-03-24T11:25:00.833 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 499.76422856300002, 2024-03-24T11:25:00.833 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.63059113300000003, 2024-03-24T11:25:00.834 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.834 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.834 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.834 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.14156", 2024-03-24T11:25:00.834 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/4087802319", 2024-03-24T11:25:00.835 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 1 2024-03-24T11:25:00.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.835 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.835 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.835 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065262+0000", 2024-03-24T11:25:00.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.836 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.836 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.836 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065262+0000", 2024-03-24T11:25:00.836 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.837 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.837 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065264+0000", 2024-03-24T11:25:00.837 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3799999999999999e-06 2024-03-24T11:25:00.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.838 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.838 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065269+0000", 2024-03-24T11:25:00.838 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4500000000000003e-06 2024-03-24T11:25:00.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.839 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.839 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065279+0000", 2024-03-24T11:25:00.839 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5959999999999993e-06 2024-03-24T11:25:00.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.840 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.840 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065289+0000", 2024-03-24T11:25:00.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0155e-05 2024-03-24T11:25:00.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.842 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.842 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065354+0000", 2024-03-24T11:25:00.842 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5376999999999994e-05 2024-03-24T11:25:00.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.065370+0000", 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5548000000000001e-05 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.694719+0000", 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.629349241 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.694839+0000", 2024-03-24T11:25:00.843 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119639 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.694971+0000", 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000131994 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695337+0000", 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00036660700000000002 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.844 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695489+0000", 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000151698 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695741+0000", 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025156199999999999 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695803+0000", 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1896000000000005e-05 2024-03-24T11:25:00.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:16:41.695854+0000", 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0989999999999998e-05 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:64308 7.b 7:de14c4e2:::benchmark_data_smithi179_78267_object64307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 197.96795950200001, 2024-03-24T11:25:00.846 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.061113733000000003, 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 64308 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861532+0000", 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.15e-07 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.848 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861551+0000", 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9168e-05 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861559+0000", 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7149999999999998e-06 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.861565+0000", 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7209999999999999e-06 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.849 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868075+0000", 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065101780000000001 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868193+0000", 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117935 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.868332+0000", 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013922699999999999 2024-03-24T11:25:00.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917732+0000", 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049399729000000003 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.917778+0000", 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6468999999999998e-05 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922502+0000", 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047234219999999997 2024-03-24T11:25:00.851 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922598+0000", 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6825999999999998e-05 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922645+0000", 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6928000000000002e-05 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:64321 7.b 7:d4b02eee:::benchmark_data_smithi179_78267_object64320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:42.871087+0000", 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 197.95840429099999, 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051668283000000002, 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 64321 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.853 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871087+0000", 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871087+0000", 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871089+0000", 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2350000000000002e-06 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871308+0000", 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021871299999999999 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871320+0000", 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2632e-05 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.871329+0000", 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7399999999999993e-06 2024-03-24T11:25:00.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915067+0000", 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043738251999999998 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915220+0000", 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015314699999999999 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.915916+0000", 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00069530899999999997 2024-03-24T11:25:00.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922454+0000", 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065381079999999999 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922491+0000", 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6823000000000003e-05 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922653+0000", 2024-03-24T11:25:00.857 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016257700000000001 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922721+0000", 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8170999999999995e-05 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:42.922755+0000", 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3575999999999998e-05 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.858 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:65602 7.b 7:d61d11e0:::benchmark_data_smithi179_78267_object65601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:44.073820+0000", 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 196.75567152400001, 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.073232816000000006, 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 65602 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.859 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073820+0000", 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073820+0000", 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073821+0000", 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.562e-06 2024-03-24T11:25:00.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073844+0000", 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2507999999999999e-05 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073856+0000", 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2463000000000001e-05 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.073864+0000", 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8390000000000007e-06 2024-03-24T11:25:00.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077258+0000", 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003393741 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077376+0000", 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011797400000000001 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077466+0000", 2024-03-24T11:25:00.862 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0828999999999994e-05 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143415+0000", 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065948735999999994 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143655+0000", 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000239349 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.144072+0000", 2024-03-24T11:25:00.863 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00041780499999999999 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.146609+0000", 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025367800000000002 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.147052+0000", 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044323000000000002 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:65607 7.0 e104/92 7:0abd0400:::benchmark_data_smithi179_78267_object65606:head v 104'4199, mlcod=104'4197)", 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:44.077314+0000", 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 196.752177123, 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.062365505000000002, 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077314+0000", 2024-03-24T11:25:00.865 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077314+0000", 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077315+0000", 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7999999999999995e-07 2024-03-24T11:25:00.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077371+0000", 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6165000000000002e-05 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077378+0000", 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7449999999999998e-06 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077382+0000", 2024-03-24T11:25:00.868 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5759999999999999e-06 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077428+0000", 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5433999999999999e-05 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.077441+0000", 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3534e-05 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.869 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.139579+0000", 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.062137723999999998 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.139679+0000", 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100647 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:65602 7.b e104/92 ondisk, result = 0)", 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:25:00.870 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 196.75037821999999, 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.064313464000000001, 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079113+0000", 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5799999999999999e-07 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.079117+0000", 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.428e-06 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.137173+0000", 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.058056435000000003 2024-03-24T11:25:00.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.137184+0000", 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0368e-05 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143401+0000", 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062175820000000001 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143410+0000", 2024-03-24T11:25:00.873 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6459999999999999e-06 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:44.143426+0000", 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6447e-05 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:96310 7.b 7:d4fc43af:::benchmark_data_smithi179_78267_object96309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:22:13.586508+0000", 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 167.24298306599999, 2024-03-24T11:25:00.874 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052402246999999999, 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 96310 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.586508+0000", 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.586508+0000", 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.586509+0000", 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3300000000000001e-07 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.586626+0000", 2024-03-24T11:25:00.876 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011711500000000001 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.586640+0000", 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3701e-05 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.586649+0000", 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2760000000000006e-06 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.877 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.611110+0000", 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024460856 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.611218+0000", 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108484 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.611354+0000", 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013582700000000001 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.618674+0000", 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0073198120000000002 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.620912+0000", 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022381369999999999 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.634826+0000", 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013914416000000001 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.635724+0000", 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00089706699999999998 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:22:13.638910+0000", 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031867229999999998 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.880 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:170937 7.2 e124/92 7:4e2f5e24:::benchmark_data_smithi179_78267_object51743:head v 124'10703, mlcod=124'10701)", 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:32.750469+0000", 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 88.079022206000005, 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.053925592000000001, 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.750469+0000", 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.881 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.750469+0000", 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.750469+0000", 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4700000000000001e-07 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.750474+0000", 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8210000000000001e-06 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.750483+0000", 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1609999999999992e-06 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.750490+0000", 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5139999999999998e-06 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.752737+0000", 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022472550000000001 2024-03-24T11:25:00.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.752757+0000", 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0001999999999999e-05 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.804299+0000", 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051542223999999998 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.804394+0000", 2024-03-24T11:25:00.884 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5068000000000005e-05 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:170944 7.b 7:dcdcb749:::benchmark_data_smithi179_78267_object51750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:32.755491+0000", 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 88.074000514000005, 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052011139999999997, 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.885 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 170944 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755491+0000", 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755491+0000", 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.886 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755492+0000", 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5510000000000001e-06 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755497+0000", 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1760000000000001e-06 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755509+0000", 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2162e-05 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.887 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755518+0000", 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1499999999999999e-06 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755572+0000", 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.465e-05 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755692+0000", 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119758 2024-03-24T11:25:00.888 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.755803+0000", 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011091600000000001 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.804168+0000", 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048364852999999999 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.806992+0000", 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028243109999999999 2024-03-24T11:25:00.889 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.807389+0000", 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00039662600000000002 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.807455+0000", 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6402000000000006e-05 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.807502+0000", 2024-03-24T11:25:00.890 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6585000000000002e-05 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:170942 7.d e124/92 7:b5f63c75:::benchmark_data_smithi179_78267_object51748:head v 124'10646, mlcod=124'10644)", 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:32.756086+0000", 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 88.073404866999994, 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.063397695000000004, 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.891 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.756086+0000", 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.756086+0000", 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.756087+0000", 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2900000000000003e-07 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.892 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.756094+0000", 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.661e-06 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.756104+0000", 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0023e-05 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.756118+0000", 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3985000000000001e-05 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.893 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.804127+0000", 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048008955999999998 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.804155+0000", 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7960999999999999e-05 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.819433+0000", 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.015277185 2024-03-24T11:25:00.894 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.819484+0000", 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1295000000000003e-05 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:170943 7.d e124/92 7:b81bbc2d:::benchmark_data_smithi179_78267_object51749:head v 124'10647, mlcod=124'10644)", 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:32.759334+0000", 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 88.070157050999995, 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.060664510999999997, 2024-03-24T11:25:00.895 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.759334+0000", 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.759334+0000", 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.896 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.759335+0000", 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7700000000000004e-07 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.759342+0000", 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4970000000000004e-06 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.804050+0000", 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044708295000000002 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.897 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.804073+0000", 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2841999999999999e-05 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.814077+0000", 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010003715999999999 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.814121+0000", 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3769000000000001e-05 2024-03-24T11:25:00.898 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.819891+0000", 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0057698580000000001 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:32.819999+0000", 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010785699999999999 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.899 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:174128 7.0 e126/92 7:03dbd756:::benchmark_data_smithi179_78267_object54934:head v 126'11101, mlcod=126'11098)", 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:35.818304+0000", 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 85.011187368999998, 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.054055471000000001, 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818304+0000", 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.900 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818304+0000", 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818304+0000", 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2100000000000002e-07 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818308+0000", 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.552e-06 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.901 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818317+0000", 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9109999999999999e-06 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818321+0000", 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9010000000000002e-06 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818354+0000", 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2273000000000001e-05 2024-03-24T11:25:00.902 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.818367+0000", 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2985e-05 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.872309+0000", 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.053942220999999999 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:35.872359+0000", 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0207000000000002e-05 2024-03-24T11:25:00.903 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:188628 7.b 7:dcc31ade:::benchmark_data_smithi179_78267_object69434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:50.624354+0000", 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 70.205136601999996, 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.054445849999999997, 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.904 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188628 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.624354+0000", 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.624354+0000", 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.905 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.624356+0000", 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3039999999999999e-06 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.624361+0000", 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6189999999999999e-06 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.624374+0000", 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2941e-05 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.906 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.624383+0000", 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.371e-06 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.629103+0000", 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004720325 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.629367+0000", 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026380899999999999 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.907 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.629710+0000", 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0003435 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.637305+0000", 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0075943720000000003 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.637353+0000", 2024-03-24T11:25:00.908 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8059000000000002e-05 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.678646+0000", 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.041293248999999997 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.678753+0000", 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107017 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.909 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.678800+0000", 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7284000000000003e-05 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:188635 7.b 7:dcecbb00:::benchmark_data_smithi179_78267_object69441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:50.627684+0000", 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 70.201807243999994, 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.055356001000000002, 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.910 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188635 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.627684+0000", 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.627684+0000", 2024-03-24T11:25:00.911 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.627685+0000", 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.159e-06 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.627689+0000", 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.42e-06 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.912 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.627700+0000", 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0696999999999999e-05 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.627707+0000", 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2450000000000001e-06 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.634122+0000", 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064147930000000002 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.634263+0000", 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000141342 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.634397+0000", 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000133053 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.637342+0000", 2024-03-24T11:25:00.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029455800000000002 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.638017+0000", 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00067484900000000004 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.682900+0000", 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.044882845999999997 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.682991+0000", 2024-03-24T11:25:00.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1398999999999997e-05 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:50.683040+0000", 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8618000000000001e-05 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:199464 7.b 7:d9b6c49b:::benchmark_data_smithi179_78267_object80270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:01.742338+0000", 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.08715334, 2024-03-24T11:25:00.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.064094577999999999, 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 199464 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742338+0000", 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742338+0000", 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742339+0000", 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.313e-06 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742344+0000", 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5129999999999998e-06 2024-03-24T11:25:00.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742355+0000", 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1681e-05 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742364+0000", 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8449999999999992e-06 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742425+0000", 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0494000000000001e-05 2024-03-24T11:25:00.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742539+0000", 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011464300000000001 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.742652+0000", 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112661 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.752013+0000", 2024-03-24T11:25:00.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0093612309999999994 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.752023+0000", 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0369e-05 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806316+0000", 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.054292142000000002 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806381+0000", 2024-03-24T11:25:00.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5221000000000005e-05 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806432+0000", 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1465000000000003e-05 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:199468 7.b 7:d44b276d:::benchmark_data_smithi179_78267_object80274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:01.745153+0000", 2024-03-24T11:25:00.922 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.084338449999997, 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.061444248, 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 199468 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745153+0000", 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745153+0000", 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745154+0000", 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.279e-06 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745159+0000", 2024-03-24T11:25:00.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9599999999999999e-06 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745170+0000", 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1243999999999999e-05 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745178+0000", 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9549999999999992e-06 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.746414+0000", 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0012362549999999999 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.746517+0000", 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010293099999999999 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.746622+0000", 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000104557 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.752026+0000", 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054046780000000004 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.753432+0000", 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001405744 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806484+0000", 2024-03-24T11:25:00.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.053051330000000001 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806549+0000", 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5524999999999994e-05 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806597+0000", 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7790000000000002e-05 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:199470 7.b 7:d58b1123:::benchmark_data_smithi179_78267_object80276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:01.745277+0000", 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.084214048, 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.061449565999999997, 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 199470 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745277+0000", 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745277+0000", 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745278+0000", 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3300000000000002e-07 2024-03-24T11:25:00.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745281+0000", 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2839999999999998e-06 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745289+0000", 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4829999999999999e-06 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.745296+0000", 2024-03-24T11:25:00.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1700000000000002e-06 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.749362+0000", 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040664619999999999 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.749472+0000", 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109658 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.749843+0000", 2024-03-24T11:25:00.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00037165000000000001 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.752142+0000", 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022983809999999999 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.753577+0000", 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014349149999999999 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:25:00.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806640+0000", 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.053063063000000001 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806689+0000", 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9669000000000002e-05 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806727+0000", 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7197999999999999e-05 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.934 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:199468 7.b e130/92 ondisk, result = 0)", 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:01.755169+0000", 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.074322404, 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051435060999999997, 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755169+0000", 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755169+0000", 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755169+0000", 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.11e-07 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755171+0000", 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8279999999999999e-06 2024-03-24T11:25:00.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755174+0000", 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7910000000000002e-06 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755177+0000", 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7140000000000002e-06 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806470+0000", 2024-03-24T11:25:00.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051292868999999998 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806478+0000", 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8720000000000002e-06 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806604+0000", 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125776 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:199470 7.b e130/92 ondisk, result = 0)", 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:01.755213+0000", 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.074277731000002, 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051520124, 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755213+0000", 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755213+0000", 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755214+0000", 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1e-07 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.755215+0000", 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.84e-06 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806353+0000", 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.051137705999999998 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806366+0000", 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3206e-05 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806628+0000", 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00026172499999999998 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806634+0000", 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3589999999999996e-06 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806733+0000", 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9078000000000002e-05 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.942 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:199476 7.d e130/92 7:b24cb5aa:::benchmark_data_smithi179_78267_object80282:head v 130'12414, mlcod=130'12412)", 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:01.757305+0000", 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 59.072185568999998, 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.052189418000000001, 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.757305+0000", 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:00.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.757305+0000", 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.757306+0000", 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6599999999999996e-07 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.757313+0000", 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1690000000000003e-06 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:00.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806537+0000", 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.049223462000000003 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806545+0000", 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8390000000000007e-06 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806644+0000", 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9686999999999996e-05 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.806671+0000", 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6735000000000001e-05 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.809464+0000", 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002793181 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:01.809495+0000", 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0778999999999998e-05 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.946 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.947 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:00.947 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:00.947 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:25:00.950 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:00.978 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:00.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:01.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:01.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:01.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:01.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:01.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:01.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:01.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:01.361 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:01.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:01.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:01.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:01.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:01.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:01.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:01.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:01 smithi179 ceph-mon[37907]: pgmap v233: 142 pgs: 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3151237897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:25:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:01 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:25:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:01 smithi179 ceph-mon[29608]: pgmap v233: 142 pgs: 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3151237897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:25:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:01 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:25:01.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:01.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:01.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:01.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:01.878 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:01.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:01 smithi181 ceph-mon[33998]: pgmap v233: 142 pgs: 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3151237897' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:25:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:01 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T11:25:01.923 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_1' created 2024-03-24T11:25:01.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:02.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:02.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.130 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:02.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:02.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.256 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:02.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:02.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.381 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:02.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:02.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:02.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:02.595 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-24T11:25:02.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.640 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:02.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:02.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.765 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:02.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:02.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:02.891 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:02.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:02.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:02 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T11:25:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:02 smithi181 ceph-mon[33998]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:25:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:02 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:02.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T11:25:02.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[29608]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:25:02.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:02.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:02.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T11:25:02.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[37907]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T11:25:02.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:02.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:02 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:02.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.022 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:03.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:03.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:03.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.148 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:03.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:03.249 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:03.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:03.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:03.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:03.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.532 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:03.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:03.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:25:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:25:03.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:03.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:25:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:03.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:03.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:03.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:03.843 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:03 smithi179 ceph-mon[29608]: pgmap v235: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:25:03.843 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:03 smithi179 ceph-mon[29608]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:25:03.844 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:03 smithi179 ceph-mon[29608]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:25:03.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:03 smithi179 ceph-mon[37907]: pgmap v235: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:25:03.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:03 smithi179 ceph-mon[37907]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:25:03.844 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:03 smithi179 ceph-mon[37907]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:25:03.844 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:25:03.862 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:25:03.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:25:03.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:03.900 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:25:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:03 smithi181 ceph-mon[33998]: pgmap v235: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:25:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:03 smithi181 ceph-mon[33998]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:25:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:03 smithi181 ceph-mon[33998]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T11:25:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:03.921 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:03.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:03.923 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:25:03.945 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:25:03.965 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:25:03.985 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:25:03.986 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:25:03.986 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:25:03.987 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:25:03.987 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:25:03.987 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:25:03.987 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:25:03.987 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:25:03.987 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:25:04.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:04.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:04.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.178 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:04.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:04.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.304 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:04.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:04.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:04.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:04.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.560 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:04.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:04.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.685 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:04.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:04.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:04.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:04 smithi181 ceph-mon[33998]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:25:04.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3256921848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:04.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:04 smithi181 ceph-mon[33998]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:25:04.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[29608]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3256921848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[29608]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[37907]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3256921848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:04.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:04 smithi179 ceph-mon[37907]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T11:25:04.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:04.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:05.046 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.072 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:05.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:05.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:05.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:05.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:25:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:25:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:25:05.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.328 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:05.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:05.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.458 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:05.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:05.559 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.583 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:05.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:05.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:05.708 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:05.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:05.810 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.835 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:05.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:05.853 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:05 smithi179 ceph-mon[29608]: pgmap v239: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:05.853 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:05 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:25:05.853 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:05 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:25:05.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:05 smithi179 ceph-mon[37907]: pgmap v239: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:05.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:05 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:25:05.854 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:05 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:25:05.893 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:25:05.894 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:25:05.894 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:25:05.894 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:25:05.894 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:25:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:05 smithi181 ceph-mon[33998]: pgmap v239: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:05 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:25:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:05 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:25:05.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:05.964 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:05.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:06.066 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:06.086 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:25:06.092 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:06.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.218 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:06.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:06.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.343 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:06.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:06.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:06.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:06.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:06.602 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:06.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:06.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:06.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:06.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/788148892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:06 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/788148892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:25:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:06 smithi181 ceph-mon[33998]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:25:06.943 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/788148892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:06.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:06.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/788148892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:25:06.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[37907]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:25:06.944 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/788148892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:06.944 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:06.944 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/788148892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:25:06.944 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:06 smithi179 ceph-mon[29608]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T11:25:06.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:06.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:06.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:06.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:07.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:07.110 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:07.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:07.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:07.236 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:07.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:07.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:25:07.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:07.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:07.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:07.411 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:07.412 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:25:07.454 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:25:07.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-24T11:25:07.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:07.521 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:07.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:07.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:07.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:07.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:07.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:07.780 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:07.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:07.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:07.884 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:07 smithi179 ceph-mon[29608]: pgmap v241: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:07.885 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:07 smithi179 ceph-mon[37907]: pgmap v241: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:07 smithi181 ceph-mon[33998]: pgmap v241: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:25:07.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:07.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:08.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:08.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:08.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:08.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:08.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:08.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:08.299 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:08.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:08.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:08.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:08.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:08.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.687 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:08.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:08.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:08.816 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:08.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:08.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:08.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:08.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:25:09.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:09.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:09.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:09.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:09.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:09.331 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:09.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:09.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:09.559 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.583 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:09.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:09.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.709 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:09.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:09.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:09 smithi179 ceph-mon[29608]: pgmap v242: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-24T11:25:09.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:09 smithi179 ceph-mon[29608]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:25:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:09 smithi179 ceph-mon[37907]: pgmap v242: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-24T11:25:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:09 smithi179 ceph-mon[37907]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:25:09.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:09.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:09 smithi181 ceph-mon[33998]: pgmap v242: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-24T11:25:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:09 smithi181 ceph-mon[33998]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T11:25:09.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:09.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:09.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:10.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.092 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:10.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:10.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:10.221 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:10.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:10.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:10.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:10.476 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:10.579 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.604 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:10.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:10.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:10.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:10.822 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:25:10.823 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:25:10.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:10.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:10.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:10.989 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:10.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:11.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:11.094 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:11.098 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:11.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:11.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:11.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:11.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:11.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:11.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:11.371 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:11.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:11.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:11.496 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:11.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:11.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:11.620 INFO:teuthology.orchestra.run.smithi179.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-24T11:25:11.625 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:11.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:11.659 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:25:11.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:11.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:11.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:11.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:11.882 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:11.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:11 smithi181 ceph-mon[33998]: pgmap v244: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-03-24T11:25:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:11 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2572796908' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:25:11.946 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:11 smithi179 ceph-mon[29608]: pgmap v244: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-03-24T11:25:11.946 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2572796908' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:25:11.947 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:11 smithi179 ceph-mon[37907]: pgmap v244: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-03-24T11:25:11.947 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:11 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2572796908' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:25:11.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:12.007 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:12.111 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:12.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:12.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:12.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:12.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.393 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:12.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:12.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:12.495 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:12.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:12.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.649 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:12.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:12.751 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.775 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:12.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:12.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:12.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:12.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2572796908' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:25:12.901 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:12 smithi179 ceph-mon[29608]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:25:12.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:12 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2572796908' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:25:12.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:12 smithi179 ceph-mon[37907]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:25:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:12.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:12 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2572796908' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:25:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:12 smithi181 ceph-mon[33998]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T11:25:13.003 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:13.031 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:13.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:13.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:13.263 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:13.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.289 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:13.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:13.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:13.413 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:13.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:13.543 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:13.619 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:25:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:25:13.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.674 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:13.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:13.717 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:13 smithi179 ceph-mon[29608]: pgmap v246: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-24T11:25:13.717 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:13 smithi179 ceph-mon[29608]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:25:13.717 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:13 smithi179 ceph-mon[37907]: pgmap v246: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-24T11:25:13.717 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:13 smithi179 ceph-mon[37907]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:25:13.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:13.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:13 smithi181 ceph-mon[33998]: pgmap v246: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-24T11:25:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:13 smithi181 ceph-mon[33998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:25:13.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:13.926 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-24T11:25:13.929 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T11:25:13.929 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T11:25:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:13.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:13.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:13.980 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:hints = 1 2024-03-24T11:25:13.980 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T11:25:13.980 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object prefix: benchmark_data_smithi179_105658 2024-03-24T11:25:13.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:25:13.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 0 1 1 0 0 0 - 0 2024-03-24T11:25:14.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:14.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:14.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:14.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:14.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:14.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:14.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:14.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:14.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:14.455 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:14.495 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:14.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:14.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:14.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:14.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:14 smithi179 ceph-mon[29608]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:25:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:14 smithi179 ceph-mon[37907]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:25:14.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:14.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:14.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:14.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:14 smithi181 ceph-mon[33998]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T11:25:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:14.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:14.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:14.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 1 16 3589 3573 223.237 223.312 0.00478845 0.00446308 2024-03-24T11:25:15.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:15.101 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:15.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:15.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:15.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:25:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:25:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:25:15.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:15.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:15.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:15.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:15.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:15.489 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:15.538 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:15.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:15.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:15.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:15.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:15 smithi179 ceph-mon[29608]: pgmap v248: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:25:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:15 smithi179 ceph-mon[37907]: pgmap v248: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:25:15.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:15.819 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:15.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:15 smithi181 ceph-mon[33998]: pgmap v248: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:25:15.921 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:15.973 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:15.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:15.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 2 16 6923 6907 215.797 208.375 0.00453619 0.00462328 2024-03-24T11:25:16.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:16.119 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:16.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:16.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:16.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:16.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:16.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:16.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:16.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:16.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:16.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:16.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:16.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:16.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:16.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:16.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:16.756 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:16.767 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:25:16.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:16.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:16.814 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:25:16.859 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:25:16.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:16.900 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:25:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:16.933 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:16.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:16.938 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:25:16.980 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:25:16.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 3 16 9609 9593 199.82 167.875 0.00351342 0.00499654 2024-03-24T11:25:17.027 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:25:17.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:17.072 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:17.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:17.074 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:25:17.082 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:25:17.175 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:17.216 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:17.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:17.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:17.355 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:17.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:17.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:17.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:17.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:17.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:17.634 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:17.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:17.650 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:25:17.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:17.779 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:17.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:17.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:17 smithi181 ceph-mon[33998]: pgmap v249: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-03-24T11:25:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:17 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/380742837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:17 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2468111898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:25:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:17 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:25:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:17 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:17.922 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:17.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[29608]: pgmap v249: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/380742837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2468111898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[37907]: pgmap v249: 158 pgs: 158 active+clean; 582 KiB data, 999 MiB used, 714 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/380742837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2468111898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:25:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:17 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:17.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 4 16 12334 12318 192.44 170.312 0.00557192 0.00518885 2024-03-24T11:25:18.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:18.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:18.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:18.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:18.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:18.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:18.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:18.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:18.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:18.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:18.471 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:18.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:18.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:18.608 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:18.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:18.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:18.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:18.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:18.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:18.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:18.875 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:18.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:18.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:18.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:18 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:18 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:25:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:18 smithi181 ceph-mon[33998]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:25:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:18 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:25:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:18 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:25:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:25:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[29608]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:25:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:25:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:25:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:25:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[37907]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T11:25:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:25:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:18 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:25:18.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:18.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 5 16 14707 14691 183.611 148.312 0.00229927 0.00540788 2024-03-24T11:25:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:19.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:19.103 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:19.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:19.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:19.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:19.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:19.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:19.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:19.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:19.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.673 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:19.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:19.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:19.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[29608]: pgmap v251: 158 pgs: 158 active+clean; 304 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 745 op/s 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[29608]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[37907]: pgmap v251: 158 pgs: 158 active+clean; 304 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 745 op/s 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:25:19.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[37907]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:25:19.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:25:19.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:19 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:25:19.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:19.952 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:19.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:19.982 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 6 16 16575 16559 172.466 116.75 0.00172689 0.0057584 2024-03-24T11:25:20.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:20.101 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:20.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:20.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:19 smithi181 ceph-mon[33998]: pgmap v251: 158 pgs: 158 active+clean; 304 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 745 op/s 2024-03-24T11:25:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:19 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:25:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:19 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:25:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:19 smithi181 ceph-mon[33998]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T11:25:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:19 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:25:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:19 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:25:20.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:20.248 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:20.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:20.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:20.394 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:20.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:20.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:20.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:20.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:20.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:20.673 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:20.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:20.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:20.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:20.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:20.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:20.959 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:20.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:20.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:25:20.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:25:20.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[29608]: 5.0 scrub starts 2024-03-24T11:25:20.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[29608]: 5.0 scrub ok 2024-03-24T11:25:20.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:25:20.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:25:20.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[37907]: 5.0 scrub starts 2024-03-24T11:25:20.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:20 smithi179 ceph-mon[37907]: 5.0 scrub ok 2024-03-24T11:25:20.982 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 7 15 18840 18825 168.059 141.625 0.00952561 0.00594389 2024-03-24T11:25:21.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:21.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:21.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:20 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:25:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:20 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:25:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:20 smithi181 ceph-mon[33998]: 5.0 scrub starts 2024-03-24T11:25:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:20 smithi181 ceph-mon[33998]: 5.0 scrub ok 2024-03-24T11:25:21.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:21.231 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:21.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:21.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:21.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:21.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:21.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:25:21.516 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:25:21.517 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:25:21.517 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:25:21.517 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:25:21.517 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:25:21.524 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:21.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:21.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:21.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:21.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:21.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:21.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:21.804 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:21.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:21.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:21.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:21.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:21.945 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:25:21.945 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:25:21.945 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[37907]: pgmap v253: 158 pgs: 158 active+clean; 833 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 846 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-03-24T11:25:21.945 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:25:21.945 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:25:21.946 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:25:21.946 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:25:21.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:25:21.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:25:21.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[29608]: pgmap v253: 158 pgs: 158 active+clean; 833 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 846 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-03-24T11:25:21.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:25:21.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:25:21.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:25:21.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:21 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:25:21.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 8 16 21175 21159 165.285 145.875 0.00478778 0.00604259 2024-03-24T11:25:22.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:22.086 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:22.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:22.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:21 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:25:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:21 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:25:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:21 smithi181 ceph-mon[33998]: pgmap v253: 158 pgs: 158 active+clean; 833 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 846 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-03-24T11:25:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:21 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:25:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:21 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:25:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:21 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:25:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:21 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:25:22.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:22.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:22.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:22.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:22.373 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:22.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:22.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:22.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:22.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:22.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:22.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:22.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:22.699 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:25:22.729 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:25:22.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:22.823 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:22.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:22.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:22.959 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:25:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:25:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:25:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:25:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:25:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 3.8 scrub starts 2024-03-24T11:25:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[29608]: 3.8 scrub ok 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 3.8 scrub starts 2024-03-24T11:25:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:22 smithi179 ceph-mon[37907]: 3.8 scrub ok 2024-03-24T11:25:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:22.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:22.981 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 9 16 23573 23557 163.57 149.875 0.00414399 0.00610799 2024-03-24T11:25:23.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:23.110 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:23.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:25:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:25:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:25:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:25:23.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:25:23.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:25:23.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 3.8 scrub starts 2024-03-24T11:25:23.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:22 smithi181 ceph-mon[33998]: 3.8 scrub ok 2024-03-24T11:25:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:25:23.190 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:25:23.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-24T11:25:23.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:23.242 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:23.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:23.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:23.391 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:23.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:23.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:23.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:23.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:23.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:23.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:25:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:25:23.669 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:23.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:23.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:23.799 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:23.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:23.831 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:25:23.831 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:25:23.831 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:25:23.832 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:25:23.832 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:25:23.853 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:23.854 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:25:23.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:23.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:23.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:23.982 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 10 16 25591 25575 159.813 126.125 0.00406358 0.00625061 2024-03-24T11:25:24.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:24.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:24.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:24.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:23 smithi181 ceph-mon[33998]: pgmap v254: 158 pgs: 158 active+clean; 833 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-24T11:25:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:23 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:25:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:23 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:25:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:23 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:25:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:23 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:25:24.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:24.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[29608]: pgmap v254: 158 pgs: 158 active+clean; 833 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-24T11:25:24.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:25:24.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:25:24.181 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:25:24.181 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:25:24.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[37907]: pgmap v254: 158 pgs: 158 active+clean; 833 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-24T11:25:24.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:25:24.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:25:24.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:25:24.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:23 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:25:24.214 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:24.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:24.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:24.351 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:24.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:24.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:24.491 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:24.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:24.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:24.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:24.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:24.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:24.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:24.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:24.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:24.897 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:24.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:24.982 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 11 16 27663 27647 157.057 129.5 0.00248707 0.0063605 2024-03-24T11:25:24.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:25.036 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:25.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:25.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:24 smithi179 ceph-mon[29608]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:25:25.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:24 smithi179 ceph-mon[29608]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:25:25.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:25.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:24 smithi179 ceph-mon[37907]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:25:25.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:24 smithi179 ceph-mon[37907]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:25:25.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:25.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:24 smithi181 ceph-mon[33998]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T11:25:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:24 smithi181 ceph-mon[33998]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T11:25:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:25.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:25.275 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.300 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:25.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:25.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:25.443 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:25:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:25:25.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:25.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:25.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:25.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:25.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.837 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:25.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:25.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:25.972 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:25.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:25.982 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 12 16 29331 29315 152.656 104.25 0.00600901 0.00654538 2024-03-24T11:25:26.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:26.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:26.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:26.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:25 smithi181 ceph-mon[33998]: pgmap v257: 158 pgs: 158 active+clean; 1.5 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 203 MiB/s wr, 3.25k op/s 2024-03-24T11:25:26.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:25 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:25:26.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:25 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:25:26.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:25 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:25:26.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:25 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:25:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[29608]: pgmap v257: 158 pgs: 158 active+clean; 1.5 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 203 MiB/s wr, 3.25k op/s 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[37907]: pgmap v257: 158 pgs: 158 active+clean; 1.5 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 203 MiB/s wr, 3.25k op/s 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:25:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:25 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:25:26.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:26.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:26.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:26.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:26.396 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:26.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:26.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:26.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:26.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:26.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:26.664 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:26.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:26.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:26.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:26.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:26.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:26.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:26.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:26.983 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 13 16 31173 31157 149.768 115.125 0.00440044 0.00667246 2024-03-24T11:25:27.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:27.067 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:26 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:25:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:26 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:25:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:26 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:25:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:26 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:25:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:26 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:25:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:26 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:25:27.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:27.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:25:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:25:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:25:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:25:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:26 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:25:27.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:27.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:27.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.462 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:27.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:27.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:27.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:27.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:27.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:27.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:27.855 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:27.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[29608]: pgmap v258: 158 pgs: 158 active+clean; 1.5 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 725 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:25:27.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:25:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:25:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[37907]: pgmap v258: 158 pgs: 158 active+clean; 1.5 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 725 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-24T11:25:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:25:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:25:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:25:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:27 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:25:27.983 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 14 16 33036 33020 147.387 116.438 0.0096974 0.00677872 2024-03-24T11:25:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:27.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:27.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:28.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:28.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:28.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:28.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:27 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:25:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:27 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:25:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:27 smithi181 ceph-mon[33998]: pgmap v258: 158 pgs: 158 active+clean; 1.5 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 725 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-24T11:25:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:27 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:25:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:27 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:25:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:27 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:25:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:27 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:25:28.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:28.273 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:28.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:28.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:28.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:28.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:28.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:28.549 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:28.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:28.651 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:28.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:28.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:28.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:28.830 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:28.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:28.918 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:28.932 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:28.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:25:28.983 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 15 16 34933 34917 145.465 118.562 0.00762393 0.00686898 2024-03-24T11:25:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:28.995 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:28.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:29.022 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:25:29.022 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:25:29.022 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:25:29.022 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:25:29.023 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:28 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:25:29.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:29.128 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:29.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:29.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:28 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:25:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:28 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:25:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:28 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:25:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:28 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:25:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:28 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:25:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:28 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:25:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:28 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:25:29.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:29.263 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:29.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:29.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:29.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:29.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:29.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:29.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:29.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:29.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:29.688 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:29.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:29.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:29.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:29.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:29.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:29.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[29608]: pgmap v259: 158 pgs: 158 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-24T11:25:29.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:25:29.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:25:29.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:25:29.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[37907]: pgmap v259: 158 pgs: 158 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-24T11:25:29.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:25:29.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:25:29.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:29 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:25:29.964 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:29.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:29.983 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 16 16 36884 36868 143.994 121.938 0.0111971 0.00693991 2024-03-24T11:25:30.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:30.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:30.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:29 smithi181 ceph-mon[33998]: pgmap v259: 158 pgs: 158 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-24T11:25:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:29 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:25:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:29 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:25:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:29 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:25:30.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:30.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:30.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:30.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:30.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:30.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:30.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:30.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:30.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:30.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:30.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:30.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:30.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:30.793 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:30.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:30.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:30.936 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:30.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:30.983 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 17 16 38833 38817 142.689 121.812 0.0145885 0.00700317 2024-03-24T11:25:31.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:30 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:25:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:30 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:25:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:30 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:25:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:30 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:25:31.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:31.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:31.174 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:25:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:25:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:25:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:25:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:25:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:25:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:25:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:30 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:25:31.212 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:31.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:31.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:31.351 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:31.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:31.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:31.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:31.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:31.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:31.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:31.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:31.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:31.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:31.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:31.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:31.894 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:31 smithi179 ceph-mon[29608]: pgmap v260: 158 pgs: 158 active+clean; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-24T11:25:31.894 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:31 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:25:31.894 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:31 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:25:31.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:31 smithi179 ceph-mon[37907]: pgmap v260: 158 pgs: 158 active+clean; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-24T11:25:31.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:31 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:25:31.894 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:31 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:25:31.906 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:31.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:31.983 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 18 16 40683 40667 141.184 115.625 0.00670575 0.00707823 2024-03-24T11:25:32.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:32.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:32.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:32.143 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:32.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:31 smithi181 ceph-mon[33998]: pgmap v260: 158 pgs: 158 active+clean; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-24T11:25:32.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:31 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:25:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:31 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:25:32.188 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:32.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:32.240 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:32.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:32.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:32.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:32.361 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:25:32.406 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:25:32.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:32.452 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:25:32.467 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:32.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:32.502 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:25:32.545 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:25:32.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:32.589 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:25:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:32.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:32.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:32.627 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:25:32.663 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:25:32.671 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:25:32.671 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:25:32.672 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:25:32.672 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:25:32.672 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:25:32.672 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:25:32.672 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:25:32.672 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:25:32.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:32.729 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:32.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:32.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:32.872 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:32.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:25:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:25:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1210090065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:32.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:25:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:25:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1210090065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:32.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:32.983 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 19 16 42163 42147 138.622 92.5 0.00783303 0.00720757 2024-03-24T11:25:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:33.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:33.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:33.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:33.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:32 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:25:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:32 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:25:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:32 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1210090065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:33.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:33.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:33.376 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:33.414 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:33.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:33.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:33.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:25:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:25:33.657 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:33.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:33.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:33.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:33.966 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:33.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:33.984 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:25:33.984071+0000 min lat: 0.00127075 max lat: 0.16087 avg lat: 0.00730513 2024-03-24T11:25:33.984 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:25:33.984 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 20 15 43789 43774 136.774 101.688 0.0147844 0.00730513 2024-03-24T11:25:34.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:34.100 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:34.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:33 smithi181 ceph-mon[33998]: pgmap v261: 158 pgs: 158 active+clean; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 607 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-24T11:25:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:33 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:25:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:33 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:25:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:33 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:25:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:33 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:25:34.159 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[29608]: pgmap v261: 158 pgs: 158 active+clean; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 607 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-24T11:25:34.159 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:25:34.159 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:25:34.159 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:25:34.159 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:25:34.159 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[37907]: pgmap v261: 158 pgs: 158 active+clean; 2.1 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 607 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-24T11:25:34.160 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:25:34.160 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:25:34.160 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:25:34.160 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:33 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:25:34.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:34.243 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:34.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:34.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:34.378 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:34.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:34.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:34.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:34.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:34.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:34.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:34.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:34.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:34.799 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:34.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:34.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:34.932 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:34.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:25:34.976 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:25:34.977 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:25:34.977 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:34.977 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:34 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:25:34.977 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:34 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:25:34.977 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:34.977 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:34 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:25:34.977 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:34 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:25:34.984 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 21 16 45419 45403 135.102 101.812 0.0220772 0.00739524 2024-03-24T11:25:35.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:35.075 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:35.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:35.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:34 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:25:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:34 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:25:35.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:35.222 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:35.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:35.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:35.365 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:35.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:35.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:25:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:25:35.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:35.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:35.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:35.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:35.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:35.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:35.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:35.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:35.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:35.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:35.922 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:35.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:35.923 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: pgmap v262: 158 pgs: 158 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1018 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: 4.0 scrub starts 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: 4.0 scrub ok 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: pgmap v262: 158 pgs: 158 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1018 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: 4.0 scrub starts 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: 4.0 scrub ok 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:35 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:35.985 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 22 16 46978 46962 133.386 97.4375 0.0128905 0.00749108 2024-03-24T11:25:36.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:36.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:36.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: pgmap v262: 158 pgs: 158 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1018 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: 4.0 scrub starts 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: 4.0 scrub ok 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:35 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:36.163 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:36.206 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:36.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:36.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:36.344 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:36.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:36.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:36.484 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:36.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:36.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:36.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:36.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:36.729 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:36.770 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:36.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:36.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:36.912 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:36.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:36.985 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 23 16 48572 48556 131.918 99.625 0.00282056 0.00756873 2024-03-24T11:25:37.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:37.058 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:37.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:25:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:25:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:25:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:25:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:25:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:25:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:25:37.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:25:37.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:36 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:37.159 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:37.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:37.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:37.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:25:37.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:36 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:37.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:37.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:37.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:37.434 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:37.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:37.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:37.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:37.614 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:37.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:37.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:37.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:25:37.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:37.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:37.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:37.889 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:37.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:37.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[29608]: pgmap v263: 158 pgs: 158 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-24T11:25:37.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:25:37.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:25:37.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:25:37.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:25:37.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:25:37.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:25:37.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[37907]: pgmap v263: 158 pgs: 158 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-24T11:25:37.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:25:37.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:25:37.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:25:37.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:25:37.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:25:37.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:37 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:25:37.985 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 24 16 50087 50071 130.367 94.6875 0.00933738 0.00766506 2024-03-24T11:25:37.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:38.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:38.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:38.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:38.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:38.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:38.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:37 smithi181 ceph-mon[33998]: pgmap v263: 158 pgs: 158 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-24T11:25:38.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:37 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:25:38.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:37 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:25:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:37 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:25:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:37 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:25:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:37 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:25:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:37 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:25:38.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:38.285 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:38.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:38.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:38.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:38.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:38.525 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:38.579 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:38.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:25:38.584 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:25:38.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-24T11:25:38.681 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:38.747 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:38.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:38.819 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:25:38.850 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:38.907 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:38.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:25:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:25:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:25:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3635741602' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 6.1b scrub starts 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[29608]: 6.1b scrub ok 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:25:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3635741602' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 6.1b scrub starts 2024-03-24T11:25:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:38 smithi179 ceph-mon[37907]: 6.1b scrub ok 2024-03-24T11:25:38.985 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 25 16 51552 51536 128.814 91.5625 0.012652 0.00775842 2024-03-24T11:25:39.008 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:39.049 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:39.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:39.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:25:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:25:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:25:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:25:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3635741602' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:25:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:25:39.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:25:39.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:25:39.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:25:39.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 6.1b scrub starts 2024-03-24T11:25:39.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:38 smithi181 ceph-mon[33998]: 6.1b scrub ok 2024-03-24T11:25:39.198 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:39.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:39.254 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:39.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:39.346 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:39.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:39.447 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:39.494 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:39.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:39.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:39.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:39.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:39.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:39.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:39.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:39.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:39.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:39.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:39.925 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:39.986 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 26 16 53118 53102 127.624 97.875 0.00798522 0.00783043 2024-03-24T11:25:39.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:40.042 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:40.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:40.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:40.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: pgmap v264: 158 pgs: 158 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: 4.1d scrub starts 2024-03-24T11:25:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:39 smithi181 ceph-mon[33998]: 4.1d scrub ok 2024-03-24T11:25:40.191 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:40.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: pgmap v264: 158 pgs: 158 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:25:40.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: 4.1d scrub starts 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[29608]: 4.1d scrub ok 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: pgmap v264: 158 pgs: 158 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:25:40.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:25:40.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T11:25:40.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:25:40.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:25:40.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: 4.1d scrub starts 2024-03-24T11:25:40.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:39 smithi179 ceph-mon[37907]: 4.1d scrub ok 2024-03-24T11:25:40.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:40.333 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:40.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:40.435 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:40.469 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:40.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:40.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:40.625 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:40.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:40.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:40.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:40.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:40.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:40.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:40.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:40.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:40.986 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 27 16 54781 54765 126.747 103.938 0.016397 0.00788457 2024-03-24T11:25:40.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:41.020 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:41.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:41.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:41.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:41.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:41.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:41.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:41.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:41.390 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: pgmap v268: 158 pgs: 158 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 5.4 deep-scrub starts 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 5.4 deep-scrub ok 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:25:41.391 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: pgmap v268: 158 pgs: 158 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 5.4 deep-scrub starts 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 5.4 deep-scrub ok 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:25:41.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:41 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:25:41.400 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:41.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: pgmap v268: 158 pgs: 158 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 5.4 deep-scrub starts 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 5.4 deep-scrub ok 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:25:41.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:41 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:25:41.459 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:41.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:41.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:41.596 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:41.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:41.698 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:41.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:41.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:41.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:41.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:41.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:41.981 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:41.986 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 28 16 56325 56309 125.664 96.5 0.0102878 0.00795351 2024-03-24T11:25:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.022 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:42.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:42.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:42.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:42.206 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:25:42.206 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:25:42.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.290 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:42.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:42.310 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:42 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:25:42.310 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:42 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:25:42.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:42 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:25:42.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:42 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:25:42.391 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:42 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:25:42.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:42 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:25:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.433 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:42.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:42.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:42.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:25:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:42.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:42.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:42.714 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:42.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:42.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:42.947 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:42.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:42.988 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 29 16 57789 57773 124.479 91.5 0.0203915 0.00802809 2024-03-24T11:25:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:42.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:43.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:43.209 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:43.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:43.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:43.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:43.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:43 smithi181 ceph-mon[33998]: 6.2 scrub starts 2024-03-24T11:25:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:43 smithi181 ceph-mon[33998]: 6.2 scrub ok 2024-03-24T11:25:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:43 smithi181 ceph-mon[33998]: pgmap v269: 158 pgs: 158 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:25:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:43 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:25:43.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:43 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:25:43.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:25:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:25:43.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:43.463 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[29608]: 6.2 scrub starts 2024-03-24T11:25:43.463 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[29608]: 6.2 scrub ok 2024-03-24T11:25:43.463 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[29608]: pgmap v269: 158 pgs: 158 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:25:43.463 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:25:43.463 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:25:43.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[37907]: 6.2 scrub starts 2024-03-24T11:25:43.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[37907]: 6.2 scrub ok 2024-03-24T11:25:43.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[37907]: pgmap v269: 158 pgs: 158 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:25:43.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:25:43.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:43 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:25:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:43.489 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:43.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:43.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:43.626 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:43.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:43.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:43.758 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:43.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:43.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:43.903 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:43.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:43.988 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 30 16 59294 59278 123.465 94.0625 0.0113421 0.00809486 2024-03-24T11:25:44.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:44.045 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:44.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.180 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:44.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:44.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.314 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:44.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: 1.8 deep-scrub starts 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: 1.8 deep-scrub ok 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:44 smithi181 ceph-mon[33998]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:25:44.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:44.438 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: 1.8 deep-scrub starts 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: 1.8 deep-scrub ok 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:44.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[37907]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: 1.8 deep-scrub starts 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: 1.8 deep-scrub ok 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:44 smithi179 ceph-mon[29608]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T11:25:44.540 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.581 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:44.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:44.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:44.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:25:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.732 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:44.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:44.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:44.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:44.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:44.989 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:44.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:44.989 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 31 16 60525 60509 121.964 76.9375 0.00285794 0.00818644 2024-03-24T11:25:45.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:45.127 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:45.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:45.230 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:45.268 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:45.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:45.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:45.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:45 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:25:45.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:45 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:25:45.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:45 smithi181 ceph-mon[33998]: pgmap v271: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-24T11:25:45.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:45 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:25:45.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:45 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:25:45.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:45 smithi181 ceph-mon[33998]: 6.5 deep-scrub starts 2024-03-24T11:25:45.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:45 smithi181 ceph-mon[33998]: 6.5 deep-scrub ok 2024-03-24T11:25:45.405 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:45.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:25:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:25:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[29608]: pgmap v271: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-24T11:25:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:25:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:25:45.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[29608]: 6.5 deep-scrub starts 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[29608]: 6.5 deep-scrub ok 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:25:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[37907]: pgmap v271: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 959 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[37907]: 6.5 deep-scrub starts 2024-03-24T11:25:45.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:45 smithi179 ceph-mon[37907]: 6.5 deep-scrub ok 2024-03-24T11:25:45.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:45.543 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:45.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:45.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:45.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:45.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:45.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:45.826 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:45.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:45.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:45.961 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:45.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:45.988 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 32 16 61930 61914 120.896 87.8125 0.0253291 0.00826674 2024-03-24T11:25:46.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:46.112 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:46.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:46.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:46.259 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:46.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:46.277 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:46 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:25:46.277 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:46 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:25:46.277 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:46 smithi179 ceph-mon[29608]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:25:46.278 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:46 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:25:46.278 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:46 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:25:46.278 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:46 smithi179 ceph-mon[37907]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:25:46.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:46.387 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:25:46.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:46.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:46.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:46 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:25:46.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:46 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:25:46.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:46 smithi181 ceph-mon[33998]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T11:25:46.426 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:25:46.468 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:25:46.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:46.508 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:25:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:46.528 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:46.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:46.545 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:25:46.581 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:25:46.624 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:25:46.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:46.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:46.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:46.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:46.665 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:25:46.669 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:25:46.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:46.804 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:46.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:46.906 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:46.957 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:46.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:46.988 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 33 15 63291 63276 119.812 85.125 0.0146563 0.00834122 2024-03-24T11:25:47.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:47.109 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:47.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:47.131 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:47.164 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:25:47.164 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:25:47.164 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:25:47.164 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 3.1d deep-scrub starts 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: 3.1d deep-scrub ok 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: pgmap v273: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 848 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2003906008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:25:47.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:25:47.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 3.1d deep-scrub starts 2024-03-24T11:25:47.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: 3.1d deep-scrub ok 2024-03-24T11:25:47.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: pgmap v273: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 848 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T11:25:47.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2003906008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:47.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:47.268 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:47.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:47.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:47.410 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:47.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:47.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:47.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:47.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:47.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 3.1d deep-scrub starts 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: 3.1d deep-scrub ok 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: pgmap v273: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 848 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T11:25:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2003906008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:25:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:47.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:47.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:47.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:47.819 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:47.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:47.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:47.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:47.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:47.988 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 34 16 64691 64675 118.86 87.4375 0.008598 0.00840755 2024-03-24T11:25:48.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:48.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:48.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:48.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:48.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:48.198 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:25:48.198 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:25:48.198 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:48.198 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:48.198 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:25:48.198 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:25:48.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:48.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:48.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:48.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:48.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:48.369 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:48.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:48.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:48.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:48.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:48.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:48.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:48.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:48 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:25:48.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:48 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:25:48.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:25:48.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:25:48.746 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:48.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:48.799 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:48.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:48.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:48.932 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:48.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:48.989 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 35 16 66053 66037 117.896 85.125 0.007726 0.00847729 2024-03-24T11:25:48.993 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:25:48.993 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:25:48.994 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:25:48.994 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:25:48.994 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:25:48.994 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:25:48.994 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:25:48.994 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:25:48.995 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:25:48.995 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:25:48.995 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:25:48.995 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:25:49.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:49.075 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:49.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:49.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:49.212 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:49.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:49.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:49.354 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:49.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[29608]: pgmap v274: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[29608]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[37907]: pgmap v274: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T11:25:49.375 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:49 smithi179 ceph-mon[37907]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:25:49.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:49.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:49.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:49.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:49.619 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:49.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:49 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:25:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:49 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:25:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:49 smithi181 ceph-mon[33998]: pgmap v274: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T11:25:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:49 smithi181 ceph-mon[33998]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T11:25:49.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:25:49.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:49.811 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:49.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:49.912 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:49.942 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:49.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:49.990 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 36 16 67371 67355 116.903 82.375 0.0088544 0.00854959 2024-03-24T11:25:50.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:50.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:50.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:50.177 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:50.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:50.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:50.308 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:50.354 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:50.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:50.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:25:50.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:25:50.434 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 4.d deep-scrub starts 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 4.d deep-scrub ok 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 1.2 deep-scrub starts 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 1.2 deep-scrub ok 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:25:50.435 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 4.d deep-scrub starts 2024-03-24T11:25:50.436 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 4.d deep-scrub ok 2024-03-24T11:25:50.436 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 1.2 deep-scrub starts 2024-03-24T11:25:50.436 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 1.2 deep-scrub ok 2024-03-24T11:25:50.436 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:25:50.436 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:50 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:25:50.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:50.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:50.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:50.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:50.619 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:50.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 4.d deep-scrub starts 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 4.d deep-scrub ok 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 1.2 deep-scrub starts 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 1.2 deep-scrub ok 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:25:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:50 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:25:50.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:50.765 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:50.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:50.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:50.905 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:50.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:50.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 37 15 68635 68620 115.881 79.0625 0.00353606 0.00862337 2024-03-24T11:25:51.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:51.046 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:51.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:51.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:51.191 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:51.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:51.224 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:25:51.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:51.330 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:51.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:51.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:51.456 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:25:51.456 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:25:51.456 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[29608]: 4.13 deep-scrub starts 2024-03-24T11:25:51.456 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[29608]: 4.13 deep-scrub ok 2024-03-24T11:25:51.456 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[29608]: pgmap v276: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 845 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:25:51.456 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3447809246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:25:51.457 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:25:51.457 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:25:51.457 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[37907]: 4.13 deep-scrub starts 2024-03-24T11:25:51.457 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[37907]: 4.13 deep-scrub ok 2024-03-24T11:25:51.457 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[37907]: pgmap v276: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 845 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:25:51.457 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3447809246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:25:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:51.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:51.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:51.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:51.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:51.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:51 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:25:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:51 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:25:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:51 smithi181 ceph-mon[33998]: 4.13 deep-scrub starts 2024-03-24T11:25:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:51 smithi181 ceph-mon[33998]: 4.13 deep-scrub ok 2024-03-24T11:25:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:51 smithi181 ceph-mon[33998]: pgmap v276: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 845 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:25:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3447809246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:25:51.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:51.748 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:51.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:51.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:51.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:51.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:51.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:51.990 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 38 15 69806 69791 114.757 73.1875 0.0151415 0.00870935 2024-03-24T11:25:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:52.013 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:52.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:52.115 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:52.154 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:52.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:52.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:52.321 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:52.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3447809246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[29608]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3447809246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:25:52.371 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:52 smithi179 ceph-mon[37907]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:25:52.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:52.464 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:52.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:52.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:52.598 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:52.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:52 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:25:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:52 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:25:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:52 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:25:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:52 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3447809246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:25:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:52 smithi181 ceph-mon[33998]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T11:25:52.700 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:25:52.747 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:25:52.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-24T11:25:52.753 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:52.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:52.854 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:52.924 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:52.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:52.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 39 16 70860 70844 113.502 65.8125 0.00767908 0.00880448 2024-03-24T11:25:52.998 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:53.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:53.068 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:25:53.068 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:70913 8.f 8:fd42d562:::benchmark_data_smithi179_105658_object70912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:53.062241+0000", 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0032141499999999998, 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00325546, 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 70913 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.069 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062241+0000", 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062241+0000", 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.070 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062242+0000", 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7090000000000001e-06 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062268+0000", 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5439000000000002e-05 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062287+0000", 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8909000000000001e-05 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.071 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062297+0000", 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0404000000000001e-05 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062421+0000", 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012415500000000001 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062586+0000", 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000164313 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.072 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.062733+0000", 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014704200000000001 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:70912 8.8 e168/150 8:1650f47e:::benchmark_data_smithi179_105658_object70911:head v 168'4584, mlcod=168'4582)", 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:53.064988+0000", 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00046713000000000001, 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00059098699999999996, 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:25:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.064988+0000", 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.064988+0000", 2024-03-24T11:25:53.074 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.064989+0000", 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.201e-06 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.065244+0000", 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025552399999999997 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.065263+0000", 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8830000000000001e-05 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.065273+0000", 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4900000000000006e-06 2024-03-24T11:25:53.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.065347+0000", 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3768000000000003e-05 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:53.065367+0000", 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0537999999999999e-05 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:25:53.077 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:25:53.078 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 2 2024-03-24T11:25:53.078 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:25:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:53.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:53.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:53.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:53.234 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:53.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:53.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:53.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:53.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:53.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:53.479 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:53 smithi179 ceph-mon[29608]: pgmap v278: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 731 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-24T11:25:53.479 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:53 smithi179 ceph-mon[37907]: pgmap v278: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 731 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-24T11:25:53.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:53.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:53.605 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:53.645 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:53.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:53.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:53 smithi181 ceph-mon[33998]: pgmap v278: 158 pgs: 158 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 731 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-24T11:25:53.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:25:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:25:53.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:53.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:53.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:53.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:53.917 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:53.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:53.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:25:53.991591+0000 min lat: 0.00127075 max lat: 0.16087 avg lat: 0.00886382 2024-03-24T11:25:53.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:25:53.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 40 16 72197 72181 112.754 83.5625 0.0092612 0.00886382 2024-03-24T11:25:54.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:54.061 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:54.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:54.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:54.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:54.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:54.293 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:54.331 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:54.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:54.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:54.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:54.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:54 smithi179 ceph-mon[29608]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:25:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:54 smithi179 ceph-mon[37907]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:25:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:54.471 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:54.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:54.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:54.606 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:54.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:54.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:25:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:54 smithi181 ceph-mon[33998]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T11:25:54.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:54.743 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:54.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:54.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:54.890 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:54.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:54.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 41 16 73400 73384 111.837 75.1875 0.0148448 0.0089369 2024-03-24T11:25:54.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:55.028 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:55.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:55.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:55.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:25:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:25:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:25:55.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:55.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:55.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:55.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:55.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:25:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:55.482 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:55 smithi179 ceph-mon[37907]: pgmap v280: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T11:25:55.482 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:55 smithi179 ceph-mon[29608]: pgmap v280: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T11:25:55.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:55.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:55 smithi181 ceph-mon[33998]: pgmap v280: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T11:25:55.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.720 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:55.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:55.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:55.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:55.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:55.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 42 16 74649 74633 111.033 78.0625 0.00676946 0.00900108 2024-03-24T11:25:55.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:55.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:56.094 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:56.134 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:56.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:56.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:56.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:56.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:56.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:56.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:56.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:56.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:56.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:56.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:56.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:56.684 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:56.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:56.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:56.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:56.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:56.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:56.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:56.957 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:56.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:56.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 43 16 75896 75880 110.263 77.9375 0.00322417 0.00905501 2024-03-24T11:25:57.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:57.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:57.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:57.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:57.230 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:57.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:57.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:57.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:57.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:57.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:57.506 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:57.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:57.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:25:57.639 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:57.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:25:57.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:57 smithi181 ceph-mon[33998]: pgmap v281: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 725 B/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:25:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:57 smithi179 ceph-mon[29608]: pgmap v281: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 725 B/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:25:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:57 smithi179 ceph-mon[37907]: pgmap v281: 158 pgs: 158 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 725 B/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:25:57.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:57.780 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:57.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:57.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:57.916 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:57.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:57.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 44 15 76981 76966 109.3 67.875 0.0834468 0.00913086 2024-03-24T11:25:58.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:58.059 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:58.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:58.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:58.204 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:58.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:58.306 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:58.345 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:58.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:58.447 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:58.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:58.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:58.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:25:58.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:58.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:25:58.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:25:58.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:58.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:25:58.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:25:58.903 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:58.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:25:58.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 45 16 78289 78273 108.686 81.6875 0.00753475 0.00919575 2024-03-24T11:25:59.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:59.046 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:59.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:59.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:25:59.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:25:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:59.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:25:59.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:25:59.290 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:25:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:59.357 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:59.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:25:59.449 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:25:59 smithi179 ceph-mon[37907]: pgmap v282: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-03-24T11:25:59.452 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:25:59 smithi179 ceph-mon[29608]: pgmap v282: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-03-24T11:25:59.458 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:59.500 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:59.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:59.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:59.650 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:59.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:59.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:25:59 smithi181 ceph-mon[33998]: pgmap v282: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-03-24T11:25:59.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:25:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:59.807 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:59.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:25:59.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:25:59.952 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:25:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:25:59.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:25:59.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 46 16 79503 79487 107.973 75.875 0.0130442 0.00925636 2024-03-24T11:26:00.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:00.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:00.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:00.173 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:26:00.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:00.211 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:26:00.218 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:00.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:00.254 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:26:00.295 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:26:00.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:00.336 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:26:00.336 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/29496546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:00.337 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/29496546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:00.364 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:00.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:00.378 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:26:00.426 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:26:00.465 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:26:00.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:00.468 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:26:00.468 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:26:00.468 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:26:00.469 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:26:00.469 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:26:00.469 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:26:00.469 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:26:00.469 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:26:00.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:00.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:00.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:00.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:00.638 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:00.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:00.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/29496546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:00.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:00.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:00.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:00.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:26:00.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:00.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:00.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:00.991 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 47 16 80716 80700 107.289 75.8125 0.00816495 0.00931443 2024-03-24T11:26:01.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:01.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:01.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:01.187 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:01.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:01.324 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:01.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.463 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:01.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:01.499 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:01 smithi179 ceph-mon[37907]: pgmap v283: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:26:01.499 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:01 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:26:01.500 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:01 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:26:01.500 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:01 smithi179 ceph-mon[29608]: pgmap v283: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:26:01.500 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:01 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:26:01.500 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:01 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:26:01.552 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:01 smithi181 ceph-mon[33998]: pgmap v283: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:26:01.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:01 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:26:01.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:01 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:26:01.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:01.579 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.596 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:01.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:01.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:01.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:01.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:01.867 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:01.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:01.992 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 48 16 81893 81877 106.586 73.5625 0.0210203 0.00937646 2024-03-24T11:26:01.997 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:01.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:02.039 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:02.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:02.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:02.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:02.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:02.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:02.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:02.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:02.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:02.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:02.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:02.560 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:02 smithi179 ceph-mon[29608]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:26:02.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:02 smithi179 ceph-mon[37907]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:26:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:02.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:02.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:02.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:02 smithi181 ceph-mon[33998]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T11:26:02.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:02.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:02.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:02.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:02.868 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:02.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:02.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:02.992 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 49 16 83104 83088 105.955 75.6875 0.00719594 0.00943255 2024-03-24T11:26:03.009 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:03.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.142 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:03.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:03.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.286 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:03.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:03.387 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.435 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:03.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:03.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:03.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:26:03.565 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:03.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:26:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:26:03.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:03.694 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:03.769 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:26:03.769 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:26:03.769 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:26:03.770 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:26:03.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:03.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:03.864 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:03 smithi179 ceph-mon[29608]: pgmap v285: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 635 B/s rd, 48 MiB/s wr, 770 op/s 2024-03-24T11:26:03.864 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:03 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:03.864 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:03.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:03 smithi179 ceph-mon[37907]: pgmap v285: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 635 B/s rd, 48 MiB/s wr, 770 op/s 2024-03-24T11:26:03.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:03 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:03.865 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:03.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:03.959 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:03.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:03.992 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 50 16 84096 84080 105.076 62 0.0149786 0.00951217 2024-03-24T11:26:04.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:04.105 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:04.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:04.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:03 smithi181 ceph-mon[33998]: pgmap v285: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 635 B/s rd, 48 MiB/s wr, 770 op/s 2024-03-24T11:26:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:03 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:04.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:04.256 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:04.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:04.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:04.401 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:04.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:04.503 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:04.546 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:04.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:04.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:04.686 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:04.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:04.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:04.788 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:04.827 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:04.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:04.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:04 smithi179 ceph-mon[29608]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:26:04.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:04.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:04 smithi179 ceph-mon[37907]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:26:04.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:04.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:04.992 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 51 16 85306 85290 104.499 75.625 0.00731039 0.009565 2024-03-24T11:26:05.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:05.097 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:05.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:04 smithi181 ceph-mon[33998]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T11:26:05.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:05.239 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:05.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:05.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:05.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:05.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:05.451 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:26:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:26:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:26:05.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:05.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:05.523 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:05.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:05.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:05.665 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:05.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:05 smithi179 ceph-mon[29608]: pgmap v287: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-24T11:26:05.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:05 smithi179 ceph-mon[37907]: pgmap v287: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-24T11:26:05.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:05.804 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:05.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:05.905 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:26:05.908 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:26:05.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-24T11:26:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:05.946 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:05.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:05.992 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 52 16 86544 86528 103.977 77.375 0.00880628 0.00961251 2024-03-24T11:26:06.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:06.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:06.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:06.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:05 smithi181 ceph-mon[33998]: pgmap v287: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-24T11:26:06.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:06.220 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:06.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:06.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:06.366 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:06.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:06.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:06.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:06.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:06.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:06.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:06.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:06.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:06.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:06.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:06.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:06.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:06.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:06.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:06.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:06.992 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 53 16 87708 87692 103.388 72.75 0.00387672 0.00966483 2024-03-24T11:26:07.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:07.046 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:07.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:07.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:07.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:07.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:07.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:07.314 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:07.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:07.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:07.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:07.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:07.559 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:07.595 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:07.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:07.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:07.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:07.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:07.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:07.863 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:07.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:07 smithi179 ceph-mon[29608]: pgmap v288: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:26:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:07 smithi179 ceph-mon[37907]: pgmap v288: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:26:07.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:07.992 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 54 16 88887 88871 102.837 73.6875 0.00869488 0.00971938 2024-03-24T11:26:08.008 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:08.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:08.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:08.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:08.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:08.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:07 smithi181 ceph-mon[33998]: pgmap v288: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:26:08.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:08.284 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:08.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:08.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:08.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:08.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:08.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:08.542 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:26:08.543 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:26:08.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:08.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:08.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:08.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:08.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:08.780 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:08.804 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:08.877 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:08.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:08.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:08.993 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 55 16 90010 89994 102.244 70.1875 0.00981309 0.00977606 2024-03-24T11:26:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:09.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:09.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:09.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:09.165 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:09.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:09.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:09.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:09.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:09.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:09.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:09.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:09.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:26:09.485 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:09.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:09.567 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:09.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:09.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:09.697 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:26:09.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:09.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:09.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:09.844 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:09.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:09.946 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[29608]: pgmap v289: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-24T11:26:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3289070930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:26:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:26:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:26:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[37907]: pgmap v289: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-24T11:26:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3289070930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:26:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:26:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:09 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:26:09.994 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 56 16 91190 91174 101.735 73.75 0.0103246 0.00982456 2024-03-24T11:26:10.000 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:10.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:10.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:10.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:09 smithi181 ceph-mon[33998]: pgmap v289: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 928 op/s 2024-03-24T11:26:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3289070930' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:26:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:09 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:26:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:09 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:26:10.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:10.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:10.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:10.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:10.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:10.545 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:10.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:10.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:10.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:10.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:10.911 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:10.955 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:10.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3289070930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:26:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[29608]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:26:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:26:10.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:26:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3289070930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:26:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[37907]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:26:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:26:10.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:10 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:26:10.993 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 57 16 92402 92386 101.279 75.75 0.00684225 0.00986911 2024-03-24T11:26:11.057 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:11.139 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:11.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:11.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:10 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:11.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:10 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3289070930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:26:11.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:10 smithi181 ceph-mon[33998]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T11:26:11.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:10 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:26:11.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:10 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:26:11.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:11.286 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:11.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:11.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:11.421 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:11.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:11.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:11.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:11.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:11.660 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:11.695 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:11.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:11.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:11.834 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:11.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:11.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:11.945 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:11 smithi179 ceph-mon[29608]: pgmap v291: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T11:26:11.945 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:11 smithi179 ceph-mon[29608]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:26:11.946 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:11 smithi179 ceph-mon[37907]: pgmap v291: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T11:26:11.946 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:11 smithi179 ceph-mon[37907]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:26:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:11.980 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:11.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:11.993 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 58 16 93662 93646 100.89 78.75 0.00630485 0.00990713 2024-03-24T11:26:12.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:12.116 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:12.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:11 smithi181 ceph-mon[33998]: pgmap v291: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T11:26:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:11 smithi181 ceph-mon[33998]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T11:26:12.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:12.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:12.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:12.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:12.392 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:12.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:12.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:12.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:12.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:12.630 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:12.672 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:12.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:12.773 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:12.811 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:12.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:12.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:12.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:12.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:12.993 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 59 16 94928 94912 100.521 79.125 0.0116012 0.00994352 2024-03-24T11:26:13.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:13.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:26:13.091 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:13.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:13.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:13.265 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:13.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:13.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:13.395 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:13.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:13.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:13.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:13.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:13.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:13.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:13.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:13.713 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:26:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:26:13.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:13.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:13.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:13.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:13.940 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:13.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:13.994 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:26:13.994807+0000 min lat: 0.00127075 max lat: 0.16087 avg lat: 0.00999885 2024-03-24T11:26:13.994 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:26:13.994 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 60 16 95996 95980 99.9566 66.75 0.0111366 0.00999885 2024-03-24T11:26:14.032 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:13 smithi179 ceph-mon[29608]: pgmap v293: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 728 op/s 2024-03-24T11:26:14.033 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:13 smithi179 ceph-mon[37907]: pgmap v293: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 728 op/s 2024-03-24T11:26:14.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:14.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:14.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:14.117 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:14.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:13 smithi181 ceph-mon[33998]: pgmap v293: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 728 op/s 2024-03-24T11:26:14.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:14.205 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:26:14.205 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:26:14.206 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:26:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:14.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:14.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:14.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:14.399 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:14.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:14.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:14.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:14.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:14.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:14.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:14.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:14.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:14.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:14.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:14.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:14.952 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:14.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:14.994 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 61 16 97137 97121 99.487 71.3125 0.0164716 0.0100467 2024-03-24T11:26:15.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:15.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:15.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:15.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:15.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:15.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:15.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:26:15.223 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:26:15.224 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-24T11:26:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:15.239 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:15.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:15.322 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:26:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:26:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:26:15.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:15.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:15.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:15.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:15.476 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:15.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:15.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:15.574 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:26:15.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:15.617 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:26:15.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:15.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:15.655 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:26:15.700 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:26:15.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:15.742 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:26:15.780 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:15.780 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:26:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:15.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:15.814 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:26:15.859 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:26:15.859 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:15 smithi179 ceph-mon[29608]: pgmap v294: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T11:26:15.859 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:15 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3574367206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:15.860 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:15 smithi179 ceph-mon[37907]: pgmap v294: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T11:26:15.860 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:15 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3574367206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:26:15.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:26:15.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:15.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:15.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:15.994 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 62 16 98239 98223 98.9932 68.875 0.00693931 0.0100971 2024-03-24T11:26:16.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.056 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:16.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:15 smithi181 ceph-mon[33998]: pgmap v294: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T11:26:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:15 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3574367206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:16.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.196 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:16.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:16.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:16.336 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:16.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:16.478 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:16.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:16.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:16.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:16.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:16.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:16.912 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:16.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:16.994 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 63 16 99373 99357 98.5468 70.875 0.012073 0.0101429 2024-03-24T11:26:17.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:17.047 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:17.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:17.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:17.188 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:17.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:17.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:17.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:17.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:17.422 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:17.468 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:17.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:17.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:17.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:17.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:17.700 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:17.744 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:17.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:17.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:17.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:17.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:17.981 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:17.995 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 64 16 100577 100561 98.1823 75.25 0.0100841 0.0101797 2024-03-24T11:26:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:18.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:18.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:18.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:17 smithi181 ceph-mon[33998]: pgmap v295: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:26:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:26:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:26:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:17 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:18.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:18.172 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[29608]: pgmap v295: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:26:18.172 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:26:18.172 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:26:18.172 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:18.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[37907]: pgmap v295: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-24T11:26:18.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:26:18.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:26:18.173 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:17 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:18.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:18.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:18.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:18.440 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:18.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:18.580 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:18.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:18.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:18.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:18.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:18.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:18 smithi179 ceph-mon[29608]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:26:18.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:18 smithi179 ceph-mon[37907]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:26:18.991 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:18.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:18.995 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 65 16 101765 101749 97.814 74.25 0.0308172 0.0102176 2024-03-24T11:26:19.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:19.131 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:19.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:18 smithi181 ceph-mon[33998]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T11:26:19.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:19.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:19.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:19.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:19.412 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:19.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:19.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:19.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:19.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:19.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:19.640 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:19.679 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:19.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:19.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:19.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:19.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:19.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:19.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:19.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:19.995 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 66 16 102870 102854 97.3784 69.0625 0.00898921 0.0102641 2024-03-24T11:26:20.046 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:20.088 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:20.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:20.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:19 smithi181 ceph-mon[33998]: pgmap v297: 158 pgs: 158 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-24T11:26:20.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:19 smithi179 ceph-mon[29608]: pgmap v297: 158 pgs: 158 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-24T11:26:20.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:19 smithi179 ceph-mon[37907]: pgmap v297: 158 pgs: 158 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-24T11:26:20.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:20.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:20.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:20.351 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:20.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:20.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:20.485 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:20.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:20.586 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:20.635 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:20.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:20.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:20.782 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:20.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:20.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:20.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:20.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:20.995 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 67 16 103882 103866 96.869 63.25 0.0128836 0.0103185 2024-03-24T11:26:21.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:21.059 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:21.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:21.197 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:21.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:21.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:21.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.454 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:21.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:21.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.595 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:21.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:21.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:21.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:21.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:21.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:21.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:21.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:21.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:21.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:21.996 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 68 15 104850 104835 96.3351 60.5625 0.0105379 0.0103761 2024-03-24T11:26:22.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:22.144 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:22.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:22.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:21 smithi181 ceph-mon[33998]: pgmap v298: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:26:22.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:21 smithi179 ceph-mon[29608]: pgmap v298: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:26:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:21 smithi179 ceph-mon[37907]: pgmap v298: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:26:22.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:22.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:22.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:22.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:22.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:22.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:22.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:22.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:22.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:22.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:22.692 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:22.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:22.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:22.843 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:22.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:22.880 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:22.945 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:22.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:22.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:23.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 69 16 106073 106057 96.0414 76.375 0.00822031 0.0104076 2024-03-24T11:26:23.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:23.154 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:23.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:23.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:23.296 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:23.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:23.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:23.453 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:23.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:23.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:23.585 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:23.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:23.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:23.732 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:23.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:23.767 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:26:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:26:23.807 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:26:23.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:23.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:23.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:23.912 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:26:23.912 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26479 7.a 7:5f28562f:::benchmark_data_smithi179_78267_object26478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.79129702799997, 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15488612299999999, 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.913 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26479 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.914 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2200000000000003e-07 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.915 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117824+0000", 2024-03-24T11:26:23.916 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0343000000000002e-05 2024-03-24T11:26:23.916 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.916 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.916 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.916 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117833+0000", 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7610000000000002e-06 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117840+0000", 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0350000000000004e-06 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.917 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121526+0000", 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003685912 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121658+0000", 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013200899999999999 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121793+0000", 2024-03-24T11:26:23.918 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134466 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272497+0000", 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15070414700000001 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272537+0000", 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9535e-05 2024-03-24T11:26:23.919 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272575+0000", 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8544999999999998e-05 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272634+0000", 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9206999999999997e-05 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.920 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272689+0000", 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4441000000000003e-05 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26483 7.2 7:427617cd:::benchmark_data_smithi179_78267_object26482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.79043241599999, 2024-03-24T11:26:23.921 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.160267666, 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26483 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.922 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118668+0000", 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.221e-06 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.923 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118691+0000", 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2310000000000002e-05 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118703+0000", 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2140999999999999e-05 2024-03-24T11:26:23.924 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118711+0000", 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6739999999999997e-06 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122939+0000", 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042282209999999999 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.925 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123051+0000", 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011178199999999999 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123184+0000", 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013356800000000001 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.926 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278443+0000", 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155258693 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278527+0000", 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4247999999999997e-05 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278810+0000", 2024-03-24T11:26:23.927 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028318000000000001 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278891+0000", 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1002999999999997e-05 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278935+0000", 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3625000000000003e-05 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.928 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26482 7.6 e99/92 7:67386a8b:::benchmark_data_smithi179_78267_object26481:head v 99'1627, mlcod=99'1625)", 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.78703181399999, 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15002405799999999, 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:26:23.929 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.930 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.930 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.930 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.930 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122069+0000", 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.057e-06 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122090+0000", 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1392000000000001e-05 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.931 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122100+0000", 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9899999999999992e-06 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122107+0000", 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0010000000000004e-06 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122268+0000", 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016068699999999999 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.932 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122287+0000", 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9508999999999999e-05 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272051+0000", 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.149763221 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272092+0000", 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1201000000000001e-05 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.933 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26485 7.6 e99/92 7:687a6aed:::benchmark_data_smithi179_78267_object26484:head v 99'1628, mlcod=99'1625)", 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.78697385300001, 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15087070799999999, 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:26:23.934 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8000000000000001e-07 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.935 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122148+0000", 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1658000000000002e-05 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122155+0000", 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8600000000000004e-06 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122160+0000", 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.288e-06 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.936 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270971+0000", 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14881098400000001 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270992+0000", 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1002e-05 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272958+0000", 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019664079999999998 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.937 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272996+0000", 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8127999999999998e-05 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26488 7.a 7:57aba21a:::benchmark_data_smithi179_78267_object26487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.78674463099998, 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.156446115, 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.938 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26488 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.939 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122356+0000", 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.153e-06 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122473+0000", 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011658399999999999 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122488+0000", 2024-03-24T11:26:23.940 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5500999999999999e-05 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122496+0000", 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1389999999999995e-06 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123136+0000", 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00064008700000000001 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271318+0000", 2024-03-24T11:26:23.941 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14818203499999999 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271479+0000", 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000160612 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.273137+0000", 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016574700000000001 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.942 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278568+0000", 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054316099999999999 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278633+0000", 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4759999999999997e-05 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278755+0000", 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012220500000000001 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.943 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278801+0000", 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5958999999999998e-05 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26489 7.3 7:c26b0ca7:::benchmark_data_smithi179_78267_object26488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.78658722, 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15559326900000001, 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.944 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26489 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122513+0000", 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.44e-07 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122535+0000", 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2070999999999999e-05 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122543+0000", 2024-03-24T11:26:23.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.763e-06 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122549+0000", 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0519999999999998e-06 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123004+0000", 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045522100000000001 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123123+0000", 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119145 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271099+0000", 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14797587000000001 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.275853+0000", 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047536339999999996 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.276560+0000", 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00070673899999999996 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278000+0000", 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014403090000000001 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278059+0000", 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9514e-05 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278106+0000", 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6307000000000002e-05 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26490 7.1 e99/92 7:8f3a7cc4:::benchmark_data_smithi179_78267_object26489:head v 99'1661, mlcod=99'1659)", 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.78606812999999, 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14983597100000001, 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.950 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123032+0000", 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5700000000000002e-07 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123062+0000", 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.006e-05 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271241+0000", 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14817881199999999 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271247+0000", 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3219999999999998e-06 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271300+0000", 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3066000000000003e-05 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271320+0000", 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0044e-05 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272827+0000", 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015071290000000001 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272867+0000", 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9981000000000002e-05 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26493 7.3 7:cc4c69e2:::benchmark_data_smithi179_78267_object26492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:26:23.954 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.785919514, 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155262276, 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26493 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123181+0000", 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9100000000000002e-07 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123215+0000", 2024-03-24T11:26:23.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3609000000000003e-05 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270946+0000", 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14773181099999999 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270967+0000", 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0211e-05 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274532+0000", 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035658159999999999 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274642+0000", 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010941 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274767+0000", 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012473399999999999 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.277988+0000", 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032216419999999998 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278115+0000", 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012678200000000001 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278340+0000", 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000224589 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278405+0000", 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5335999999999999e-05 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278442+0000", 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7444999999999998e-05 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.960 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:26479 7.a e99/92 ondisk, result = 0)", 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 305.78481993999998, 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14841649300000001, 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:26:23.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4300000000000003e-07 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124285+0000", 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8049999999999997e-06 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271371+0000", 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14708589 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271380+0000", 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9910000000000008e-06 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272564+0000", 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011840279999999999 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272570+0000", 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7880000000000003e-06 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272696+0000", 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126348 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:154432 7.6 e121/92 7:6a2d6ca3:::benchmark_data_smithi179_78267_object35238:head v 121'11411, mlcod=121'11400)", 2024-03-24T11:26:23.964 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:15.057447+0000", 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 188.85165307400001, 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14432173000000001, 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.057447+0000", 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.057447+0000", 2024-03-24T11:26:23.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.057448+0000", 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0890000000000001e-06 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.057454+0000", 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9950000000000002e-06 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.057462+0000", 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.174e-06 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.057469+0000", 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7599999999999997e-06 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.073878+0000", 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016409186999999999 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.073901+0000", 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3518000000000001e-05 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.201728+0000", 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.127827142 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:15.201768+0000", 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9864999999999999e-05 2024-03-24T11:26:23.968 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:216682 7.f e133/92 7:fb2c4783:::benchmark_data_smithi179_78267_object97488:head v 133'13580, mlcod=133'13576)", 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:19.006749+0000", 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 124.902350949, 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14211407400000001, 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006749+0000", 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006749+0000", 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006749+0000", 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4300000000000003e-07 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006756+0000", 2024-03-24T11:26:23.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.297e-06 2024-03-24T11:26:23.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.971 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:23.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006765+0000", 2024-03-24T11:26:23.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7520000000000008e-06 2024-03-24T11:26:23.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006772+0000", 2024-03-24T11:26:23.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8079999999999999e-06 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006848+0000", 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5908000000000006e-05 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.006867+0000", 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9106999999999998e-05 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.148815+0000", 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14194732600000001 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.148863+0000", 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8232999999999997e-05 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:216704 7.7 e133/92 7:ebbea3bd:::benchmark_data_smithi179_78267_object97510:head v 133'13351, mlcod=133'13347)", 2024-03-24T11:26:23.974 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:24:19.008340+0000", 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 124.900759295, 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14399137100000001, 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008340+0000", 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008340+0000", 2024-03-24T11:26:23.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008341+0000", 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2600000000000002e-07 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008349+0000", 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1990000000000006e-06 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008358+0000", 2024-03-24T11:26:23.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0610000000000002e-06 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008365+0000", 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7739999999999999e-06 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008429+0000", 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3744999999999994e-05 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.008448+0000", 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9204000000000001e-05 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.152258+0000", 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14380964199999999 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:24:19.152332+0000", 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4120000000000002e-05 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.978 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29142 8.f 8:f192749b:::benchmark_data_smithi179_105658_object29141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.761137+0000", 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.147962315000001, 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15871291300000001, 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29142 2024-03-24T11:26:23.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761137+0000", 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761137+0000", 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761138+0000", 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.176e-06 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761160+0000", 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1665e-05 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761170+0000", 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8099999999999992e-06 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761178+0000", 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6389999999999992e-06 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.765544+0000", 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043660870000000003 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.765820+0000", 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027657199999999999 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766369+0000", 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054847499999999998 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.770225+0000", 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003855851 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771695+0000", 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001470917 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919776+0000", 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148080342 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919819+0000", 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2781000000000003e-05 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919850+0000", 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1597999999999998e-05 2024-03-24T11:26:23.984 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29149 8.f 8:fb6565ee:::benchmark_data_smithi179_105658_object29148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.764256+0000", 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.144843383000001, 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15570266899999999, 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.985 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:23.986 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:23 smithi179 ceph-mon[29608]: pgmap v299: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:26:23.986 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:23 smithi179 ceph-mon[37907]: pgmap v299: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29149 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764256+0000", 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764256+0000", 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764257+0000", 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.24e-06 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764285+0000", 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7521e-05 2024-03-24T11:26:23.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764296+0000", 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1097999999999999e-05 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764304+0000", 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3140000000000004e-06 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767982+0000", 2024-03-24T11:26:23.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003677493 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768086+0000", 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010432700000000001 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768358+0000", 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027155700000000002 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771182+0000", 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028244810000000002 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.773685+0000", 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025023200000000002 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919882+0000", 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14619728900000001 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919929+0000", 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7403e-05 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919959+0000", 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9626000000000001e-05 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29152 8.0 8:05cda426:::benchmark_data_smithi179_105658_object29151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:23.992 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.766296+0000", 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.142803233999999, 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15319068, 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29152 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766296+0000", 2024-03-24T11:26:23.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766296+0000", 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766297+0000", 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1039999999999999e-06 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766318+0000", 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1009000000000002e-05 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766327+0000", 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8990000000000007e-06 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766335+0000", 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1389999999999998e-06 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766935+0000", 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00060048000000000002 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767159+0000", 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022360500000000001 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767274+0000", 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115157 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.770451+0000", 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031773249999999999 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771159+0000", 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00070825200000000004 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919372+0000", 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148212714 2024-03-24T11:26:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919436+0000", 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3460000000000006e-05 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919487+0000", 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1536000000000003e-05 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:23.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29158 8.0 8:0bc0922d:::benchmark_data_smithi179_105658_object29157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.141497053000002, 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15200013400000001, 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29158 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:23.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3099999999999996e-07 2024-03-24T11:26:24.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767625+0000", 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1222000000000001e-05 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767636+0000", 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1581e-05 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767644+0000", 2024-03-24T11:26:24.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0590000000000003e-06 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768740+0000", 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010955979999999999 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.769027+0000", 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000286643 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.769156+0000", 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012901500000000001 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771192+0000", 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020364939999999998 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774846+0000", 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036535500000000002 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919526+0000", 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14468013599999999 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919569+0000", 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3591999999999999e-05 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919603+0000", 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3312999999999998e-05 2024-03-24T11:26:24.004 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29163 8.f 8:fe0437c8:::benchmark_data_smithi179_105658_object29162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.771176+0000", 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.137923530000002, 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148882706, 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:24.005 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29163 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771176+0000", 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771176+0000", 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771178+0000", 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.734e-06 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771204+0000", 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5853000000000001e-05 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771218+0000", 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4517e-05 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771226+0000", 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9920000000000007e-06 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771722+0000", 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00049535699999999998 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771821+0000", 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9647000000000003e-05 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771942+0000", 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012074 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.773994+0000", 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020524369999999998 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774005+0000", 2024-03-24T11:26:24.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0368e-05 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919990+0000", 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14598484 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920029+0000", 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9801000000000001e-05 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:24.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920059+0000", 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.942e-05 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29164 8.0 8:0915a6c1:::benchmark_data_smithi179_105658_object29163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.771244+0000", 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.137855234, 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148498768, 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:24.011 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29164 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771244+0000", 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771244+0000", 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.012 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771245+0000", 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9100000000000004e-07 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771264+0000", 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9519000000000001e-05 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771271+0000", 2024-03-24T11:26:24.013 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1110000000000002e-06 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771277+0000", 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6289999999999998e-06 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771349+0000", 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1464999999999994e-05 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.014 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771481+0000", 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132352 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771617+0000", 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013571699999999999 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774906+0000", 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003288853 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774976+0000", 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0302999999999994e-05 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919638+0000", 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14466245899999999 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919712+0000", 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3567999999999998e-05 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919743+0000", 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1300999999999997e-05 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29174 8.b 8:d5efc351:::benchmark_data_smithi179_105658_object29173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.776551+0000", 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.132548526000001, 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15217817, 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:24.017 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29174 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776551+0000", 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:24.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776551+0000", 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776553+0000", 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7129999999999999e-06 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776574+0000", 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.137e-05 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776634+0000", 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0146999999999999e-05 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776643+0000", 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3080000000000008e-06 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776699+0000", 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6517000000000001e-05 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776801+0000", 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101848 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776924+0000", 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012302299999999999 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778264+0000", 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013400420000000001 2024-03-24T11:26:24.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778613+0000", 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034889699999999998 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.928631+0000", 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150018491 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.928687+0000", 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5424999999999999e-05 2024-03-24T11:26:24.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.928729+0000", 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2388999999999997e-05 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29177 8.0 8:0a08d517:::benchmark_data_smithi179_105658_object29176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.778949+0000", 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 58.130150143000002, 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14199739, 2024-03-24T11:26:24.023 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29177 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778949+0000", 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:26:24.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778949+0000", 2024-03-24T11:26:24.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:26:24.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:26:24.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778951+0000", 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.341e-06 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778972+0000", 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1189e-05 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778984+0000", 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.189e-05 2024-03-24T11:26:24.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778991+0000", 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3710000000000002e-06 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.779043+0000", 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1799000000000003e-05 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.779162+0000", 2024-03-24T11:26:24.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119113 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.779991+0000", 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00082908199999999997 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.781005+0000", 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010137480000000001 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.840847+0000", 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.059841547000000002 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920849+0000", 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.080002162000000002 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920908+0000", 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9199e-05 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:26:24.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920947+0000", 2024-03-24T11:26:24.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8949000000000003e-05 2024-03-24T11:26:24.030 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.030 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:24.030 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.030 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:24.030 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:24.030 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:26:24.031 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 70 16 107149 107133 95.6289 67.25 0.0140721 0.0104514 2024-03-24T11:26:24.049 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:24.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:24.152 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:24.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:23 smithi181 ceph-mon[33998]: pgmap v299: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:26:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:24.188 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:24.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:24.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:24.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:24.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:24.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:24.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:24.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:24.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:24.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:24.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:24.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:24.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:24.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:24.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:24.867 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:24.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:24.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:24.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:25.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 71 16 108251 108235 95.252 68.875 0.00516489 0.0104928 2024-03-24T11:26:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:25.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:25.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:24 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3935536055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:26:25.055 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:24 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:26:25.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:25.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:24 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3935536055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:26:25.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:24 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:26:25.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:25.131 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:26:25.135 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:24 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3935536055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:26:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:24 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:26:25.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:25.274 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:25.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.421 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:25.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:26:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:26:25.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:25.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:25.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:25.688 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:25.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:25.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:25.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[29608]: pgmap v300: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T11:26:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:26:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[29608]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:26:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[37907]: pgmap v300: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T11:26:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:26:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:25 smithi179 ceph-mon[37907]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:26:25.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:25.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:25.999 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 72 16 109509 109493 95.0211 78.625 0.00650129 0.0105192 2024-03-24T11:26:26.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:26.120 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:26.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:25 smithi181 ceph-mon[33998]: pgmap v300: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T11:26:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:25 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:25 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:26:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:25 smithi181 ceph-mon[33998]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T11:26:26.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:26.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:26.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:26.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:26.395 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:26.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:26.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:26.537 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:26.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:26.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:26.679 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:26.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:26.780 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:26.819 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:26.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:26.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:26.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:26.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:27.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 73 16 110727 110711 94.7622 76.125 0.0314056 0.010548 2024-03-24T11:26:27.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:27.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:27.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:27.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:26 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:26:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:26 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:26:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:26 smithi181 ceph-mon[33998]: 3.18 scrub starts 2024-03-24T11:26:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:26 smithi181 ceph-mon[33998]: 3.18 scrub ok 2024-03-24T11:26:27.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:26:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:26:27.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[29608]: 3.18 scrub starts 2024-03-24T11:26:27.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[29608]: 3.18 scrub ok 2024-03-24T11:26:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:26:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:26:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[37907]: 3.18 scrub starts 2024-03-24T11:26:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:26 smithi179 ceph-mon[37907]: 3.18 scrub ok 2024-03-24T11:26:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:27.231 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:27.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:27.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:26:27.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:27.368 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:27.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:27.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:27.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:27.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:27.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:27.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:27.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:27.687 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:27.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:27.786 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:27.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:27.786 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: pgmap v302: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:26:27.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:27.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:27.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:28.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 74 16 111866 111850 94.4435 71.1875 0.00812513 0.0105836 2024-03-24T11:26:28.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:28.066 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:28.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:26:28.133 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:26:28.134 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:26:28.134 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:26:28.134 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: pgmap v302: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:26:28.134 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:26:28.134 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:26:28.134 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:26:28.134 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:26:28.134 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:26:28.135 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:27 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:26:28.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: pgmap v302: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:26:28.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:26:28.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:26:28.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:26:28.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:26:28.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:26:28.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:26:28.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:26:28.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:27 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:26:28.168 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:28.206 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:28.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:28.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:28.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:28.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:28.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:28.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:28.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:28.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:28.611 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:28.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:28.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:28.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:28.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:28.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:28.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:26:28.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:28.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:28.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:29.001 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 75 15 113081 113066 94.1976 76 0.0119307 0.010611 2024-03-24T11:26:29.037 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:29.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:29.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:26:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:28 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:26:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:29.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:29.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:26:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:26:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:26:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:26:29.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:26:29.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:26:29.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:28 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:26:29.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:29.307 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:29.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:29.381 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:29.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:26:29.428 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:26:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:29.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:29.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:29.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:29.617 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:29.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:29.720 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:29.768 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:29.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:29.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:29.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:29.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[29608]: pgmap v303: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[29608]: 4.19 scrub starts 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[29608]: 4.19 scrub ok 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[37907]: pgmap v303: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:26:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:26:29.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:26:29.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:26:29.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[37907]: 4.19 scrub starts 2024-03-24T11:26:29.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:29 smithi179 ceph-mon[37907]: 4.19 scrub ok 2024-03-24T11:26:30.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 76 16 114196 114180 93.8742 69.625 0.00690913 0.0106482 2024-03-24T11:26:30.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:30.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:30.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:30.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:29 smithi181 ceph-mon[33998]: pgmap v303: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-24T11:26:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:29 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:26:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:29 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:26:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:29 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:26:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:29 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:26:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:29 smithi181 ceph-mon[33998]: 4.19 scrub starts 2024-03-24T11:26:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:29 smithi181 ceph-mon[33998]: 4.19 scrub ok 2024-03-24T11:26:30.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:30.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:30.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:30.253 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:30.309 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:30.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:30.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:30.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:30.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:30.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:30.511 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:26:30.511 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:114800 8.5 e175/150 8:a7e917dd:::benchmark_data_smithi179_105658_object114799:head v 175'7222, mlcod=175'7220)", 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:30.504794+0000", 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0050403239999999997, 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050736109999999996, 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504794+0000", 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:30.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504794+0000", 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504795+0000", 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9800000000000003e-07 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504833+0000", 2024-03-24T11:26:30.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8068000000000002e-05 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504844+0000", 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1022000000000001e-05 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504852+0000", 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4880000000000001e-06 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:30.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504904+0000", 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2256000000000002e-05 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504928+0000", 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3725000000000001e-05 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.509847+0000", 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049193509999999998 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:30.515 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:114799 8.2 8:460ab588:::benchmark_data_smithi179_105658_object114798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:30.501288+0000", 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0085470719999999993, 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0086735739999999999, 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114799 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.516 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501288+0000", 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501288+0000", 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501289+0000", 2024-03-24T11:26:30.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.471e-06 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501317+0000", 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8167e-05 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501347+0000", 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9560000000000002e-05 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501356+0000", 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1370000000000008e-06 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501434+0000", 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7914e-05 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501557+0000", 2024-03-24T11:26:30.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122886 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501689+0000", 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013272699999999999 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.507338+0000", 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056483779999999999 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.507562+0000", 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000224033 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:114801 8.2 8:46ccbf02:::benchmark_data_smithi179_105658_object114800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:30.501752+0000", 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0080825240000000007, 2024-03-24T11:26:30.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0083077589999999996, 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114801 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501752+0000", 2024-03-24T11:26:30.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501752+0000", 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501753+0000", 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9599999999999998e-07 2024-03-24T11:26:30.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501780+0000", 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7186999999999999e-05 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501791+0000", 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1355999999999999e-05 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:30.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.501800+0000", 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6729999999999998e-06 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504150+0000", 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023497499999999998 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:30.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504565+0000", 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000415478 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.504699+0000", 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013379599999999999 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.507485+0000", 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002785591 2024-03-24T11:26:30.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:30.507890+0000", 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040558100000000002 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 3 2024-03-24T11:26:30.527 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:26:30.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:30.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:30.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:30.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:30.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:30.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:30.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:30.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:30.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:30.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:31.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 77 15 115363 115348 93.6031 73 0.00538294 0.0106787 2024-03-24T11:26:31.036 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:31.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:31.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:26:31.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:26:31.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:26:31.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:26:31.097 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:26:31.098 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:30 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:26:31.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:26:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:30 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:26:31.174 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:31.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:31.276 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:31.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:26:31.315 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:31.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:31.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:31.456 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:31.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:31.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:31.608 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:31.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:31.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:31.730 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:31.740 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:31.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:31.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:31.885 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:31.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:31.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:32.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 78 16 116484 116468 93.3005 70 0.00799613 0.0107137 2024-03-24T11:26:32.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:32.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:32.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:31 smithi181 ceph-mon[33998]: pgmap v304: 158 pgs: 158 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:26:32.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:31 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:26:32.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:31 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:26:32.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:32.158 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:31 smithi179 ceph-mon[37907]: pgmap v304: 158 pgs: 158 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:26:32.158 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:31 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:26:32.158 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:31 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:26:32.161 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:31 smithi179 ceph-mon[29608]: pgmap v304: 158 pgs: 158 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:26:32.161 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:31 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:26:32.161 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:31 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:26:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:32.171 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:32.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:32.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:32.317 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:32.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:32.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:32.455 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:32.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:32.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:32.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:32.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:32.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:32.734 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:32.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:32.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:32.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:32.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:32.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:32.875 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:26:32.923 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:26:32.955 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:26:32.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:32.986 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:26:33.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 79 16 117499 117483 92.9224 63.4375 0.00684435 0.010757 2024-03-24T11:26:33.003 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:26:33.003 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:26:33.003 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[29608]: 8.3 scrub starts 2024-03-24T11:26:33.003 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:26:33.003 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:26:33.003 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.005 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:33.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:33.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:26:33.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:26:33.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[37907]: 8.3 scrub starts 2024-03-24T11:26:33.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:26:33.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:26:33.009 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:33.030 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:26:33.082 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:26:33.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:33.129 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:26:33.135 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:33.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:32 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:26:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:32 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:26:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:32 smithi181 ceph-mon[33998]: 8.3 scrub starts 2024-03-24T11:26:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:32 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:26:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:32 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:26:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:33.169 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:26:33.177 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:26:33.177 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:26:33.177 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:26:33.177 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:26:33.177 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:26:33.177 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:26:33.177 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:26:33.178 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:26:33.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:33.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:33.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:33.513 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:33.559 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:33.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:33.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:33.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:33.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:26:33.835 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:33.860 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:26:33.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: pgmap v305: 158 pgs: 158 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1624708583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:26:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2636101687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: pgmap v305: 158 pgs: 158 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1624708583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:26:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2636101687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:26:33.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:33.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:34.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:26:34.000861+0000 min lat: 0.00127075 max lat: 0.16087 avg lat: 0.0108207 2024-03-24T11:26:34.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:26:34.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 80 16 118286 118270 92.3758 49.1875 0.0188894 0.0108207 2024-03-24T11:26:34.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:34.097 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:34.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:34.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: pgmap v305: 158 pgs: 158 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-24T11:26:34.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:26:34.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1624708583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:26:34.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2636101687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:26:34.199 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:34.243 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:34.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:34.345 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:34.383 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:34.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:34.428 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:26:34.428 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:26:34.429 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-24T11:26:34.429 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:26:34.429 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:26:34.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-24T11:26:34.433 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:26:34.433 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:26:34.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:34.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:34.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:34.615 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:34.668 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:34.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:34.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:34.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:34.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:34.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:34.974 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:34.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:35.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 81 16 119063 119047 91.8349 48.5625 0.00940046 0.0108843 2024-03-24T11:26:35.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:35.083 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: 8.1 scrub starts 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2636101687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:26:35.084 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:34 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:26:35.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:35.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:26:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: 8.1 scrub starts 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2636101687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:26:35.210 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:26:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: 8.1 scrub starts 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2636101687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:26:35.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:34 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:26:35.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:35.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:35.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:35.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:35.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:35.376 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:35.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:35.425 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:35.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:26:35.478 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:35.526 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:35.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:35.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:35.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:35.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:35.763 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:35.811 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:35.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:35.828 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:35.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: pgmap v307: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: 8.3 scrub ok 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: 3.2 scrub starts 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[29608]: 3.2 scrub ok 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: pgmap v307: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:26:35.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: 8.3 scrub ok 2024-03-24T11:26:35.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:26:35.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:26:35.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: 3.2 scrub starts 2024-03-24T11:26:35.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:35 smithi179 ceph-mon[37907]: 3.2 scrub ok 2024-03-24T11:26:35.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:35.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:35.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:36.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 82 16 120123 120107 91.5228 66.25 0.00560701 0.0109141 2024-03-24T11:26:36.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:36.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:36.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:36.086 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: pgmap v307: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:26:36.086 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:26:36.086 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:26:36.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: 8.3 scrub ok 2024-03-24T11:26:36.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:26:36.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:26:36.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: 3.2 scrub starts 2024-03-24T11:26:36.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:35 smithi181 ceph-mon[33998]: 3.2 scrub ok 2024-03-24T11:26:36.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:36.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:36.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:36.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:36.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:36.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:36.451 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:36.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:36.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:36.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:26:36.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:36.634 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:36.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:36.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:36.778 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:36.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:36.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:36.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:36.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:37.000 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 83 16 121135 121119 91.1822 63.25 0.0119239 0.0109601 2024-03-24T11:26:37.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:37.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:37.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:37.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:37.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:36 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:26:37.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:36 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:26:37.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:36 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:26:37.186 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:37.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:36 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:26:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:36 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:26:37.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:36 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:26:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:36 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:26:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:36 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:26:37.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:36 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:26:37.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:37.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:37.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:37.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:26:37.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:37.488 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:37.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:37.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:37.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:37.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:37.638 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:37.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:37.726 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:26:37.727 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:26:37.790 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:37.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:37.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:37.955 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:37.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:38.001 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 84 16 121985 121969 90.7284 53.125 0.0129858 0.011017 2024-03-24T11:26:38.015 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[37907]: pgmap v308: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[37907]: 8.1 scrub ok 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[29608]: pgmap v308: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[29608]: 8.1 scrub ok 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:26:38.016 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:26:38.017 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:37 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:26:38.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:38.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:38.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:38.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:37 smithi181 ceph-mon[33998]: pgmap v308: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-24T11:26:38.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:37 smithi181 ceph-mon[33998]: 8.1 scrub ok 2024-03-24T11:26:38.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:37 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:26:38.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:37 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:26:38.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:37 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:26:38.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:37 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:26:38.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:38.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:38.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:38.324 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:38.367 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:38.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:38.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:38.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:26:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:38.512 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:38.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:38.605 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:26:38.605 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T11:26:38.605 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:26:38.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:38.659 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:38.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:38.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:38.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:38.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:38.890 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:38.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:38.953 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:38.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: 4.5 deep-scrub starts 2024-03-24T11:26:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: 4.5 deep-scrub ok 2024-03-24T11:26:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: 6.e scrub starts 2024-03-24T11:26:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:26:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: 6.e scrub ok 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: 4.5 deep-scrub starts 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: 4.5 deep-scrub ok 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: 6.e scrub starts 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: 6.e scrub ok 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:26:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:38.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:38 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:26:39.001 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 85 16 123152 123136 90.5191 72.9375 0.00887186 0.0110431 2024-03-24T11:26:39.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:39.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:39.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: 4.5 deep-scrub starts 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: 4.5 deep-scrub ok 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: 6.e scrub starts 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: 6.e scrub ok 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:38 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:26:39.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:39.234 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:39.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:39.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:39.372 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:39.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:39.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:39.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:39.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:39.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:39.648 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:39.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:39.750 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:39.875 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:39.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[29608]: pgmap v309: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[29608]: 3.7 scrub starts 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[29608]: 3.7 scrub ok 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[37907]: pgmap v309: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:26:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[37907]: 3.7 scrub starts 2024-03-24T11:26:39.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:39 smithi179 ceph-mon[37907]: 3.7 scrub ok 2024-03-24T11:26:39.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:40.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 86 16 124366 124350 90.3483 75.875 0.019806 0.0110636 2024-03-24T11:26:40.022 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:40.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:40.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:39 smithi181 ceph-mon[33998]: pgmap v309: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:26:40.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:39 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:26:40.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:39 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:26:40.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:39 smithi181 ceph-mon[33998]: 3.7 scrub starts 2024-03-24T11:26:40.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:39 smithi181 ceph-mon[33998]: 3.7 scrub ok 2024-03-24T11:26:40.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:40.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:40.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:40.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:40.307 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:40.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:40.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:40.438 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:40.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:40.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:40.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:40.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:40.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:40.675 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:40.729 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:40.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:40.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:40.887 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:40.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:40.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:41.002 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 87 16 125301 125285 89.9815 58.4375 0.0296199 0.0111088 2024-03-24T11:26:41.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:41.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:41.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:40 smithi181 ceph-mon[33998]: 6.17 deep-scrub starts 2024-03-24T11:26:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:40 smithi181 ceph-mon[33998]: 8.a scrub starts 2024-03-24T11:26:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:40 smithi181 ceph-mon[33998]: 6.17 deep-scrub ok 2024-03-24T11:26:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:40 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:26:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:40 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:26:41.162 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:41.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:41.194 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[29608]: 6.17 deep-scrub starts 2024-03-24T11:26:41.194 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[29608]: 8.a scrub starts 2024-03-24T11:26:41.194 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[29608]: 6.17 deep-scrub ok 2024-03-24T11:26:41.194 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:26:41.194 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:26:41.195 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[37907]: 6.17 deep-scrub starts 2024-03-24T11:26:41.195 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[37907]: 8.a scrub starts 2024-03-24T11:26:41.195 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[37907]: 6.17 deep-scrub ok 2024-03-24T11:26:41.195 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:26:41.195 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:40 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:26:41.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:41.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:41.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:41.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:41.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:41.554 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:26:41.586 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:26:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.588 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:41.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:41.628 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:26:41.668 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:26:41.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:41.707 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:26:41.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:41.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:41.748 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:26:41.787 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:26:41.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:41.834 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:26:41.841 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:26:41.842 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:26:41.842 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:26:41.842 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:26:41.842 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:26:41.842 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:26:41.842 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:26:41.842 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:26:41.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:41 smithi181 ceph-mon[33998]: pgmap v310: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:26:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:41 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:26:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:41 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:26:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:41 smithi181 ceph-mon[33998]: 8.7 scrub starts 2024-03-24T11:26:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3344372618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:41.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:41.977 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:41.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:42.002 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 88 16 126183 126167 89.5854 55.125 0.00943707 0.0111546 2024-03-24T11:26:42.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:42.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:42.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:42.138 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[29608]: pgmap v310: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:26:42.138 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:26:42.138 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:26:42.138 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[29608]: 8.7 scrub starts 2024-03-24T11:26:42.138 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3344372618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:42.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[37907]: pgmap v310: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:26:42.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:26:42.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:26:42.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[37907]: 8.7 scrub starts 2024-03-24T11:26:42.139 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3344372618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:42.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:42.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:42.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:42.259 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:26:42.259 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:26:42.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.259 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:126453 8.8 8:1661c108:::benchmark_data_smithi179_105658_object126452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:26:42.259 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:42.254082+0000", 2024-03-24T11:26:42.259 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0045634780000000002, 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046033949999999997, 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126453 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254082+0000", 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:42.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254082+0000", 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254084+0000", 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6810000000000001e-06 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:42.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254129+0000", 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5631999999999999e-05 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254142+0000", 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.253e-05 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254151+0000", 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2059999999999996e-06 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254202+0000", 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1128000000000002e-05 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254309+0000", 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106601 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:26:42.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.254474+0000", 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000165544 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:126454 8.9 e176/150 8:9205909b:::benchmark_data_smithi179_105658_object126453:head v 176'7769, mlcod=176'7767)", 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:42.255467+0000", 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0031781859999999999, 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033037309999999999, 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:42.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255467+0000", 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255467+0000", 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255468+0000", 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1200000000000001e-07 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255492+0000", 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3487999999999999e-05 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255507+0000", 2024-03-24T11:26:42.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5264000000000002e-05 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255515+0000", 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6879999999999999e-06 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255567+0000", 2024-03-24T11:26:42.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2040000000000003e-05 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.255586+0000", 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.912e-05 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:126443 8.0 e176/150 8:0f10e370:::benchmark_data_smithi179_105658_object126442:head v 176'7907, mlcod=176'7904)", 2024-03-24T11:26:42.268 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:42.256441+0000", 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.002204087, 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023792420000000002, 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256441+0000", 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:42.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256441+0000", 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256442+0000", 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8800000000000002e-07 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256491+0000", 2024-03-24T11:26:42.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8775e-05 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256506+0000", 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5123000000000001e-05 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256513+0000", 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1930000000000004e-06 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:42.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256575+0000", 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1877000000000004e-05 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:42.256594+0000", 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8519999999999999e-05 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 3 2024-03-24T11:26:42.272 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:26:42.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:42.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:42.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:42.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:42.522 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:42.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:42.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:42.670 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:42.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:42.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:42.810 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:42.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:42.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:42.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:42.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:43.002 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 89 16 127166 127150 89.2691 61.4375 0.00305579 0.0111885 2024-03-24T11:26:43.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:43.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:43.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:43.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:43.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: pgmap v311: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 5.17 deep-scrub starts 2024-03-24T11:26:43.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:42 smithi181 ceph-mon[33998]: 5.17 deep-scrub ok 2024-03-24T11:26:43.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:26:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:26:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:26:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: pgmap v311: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 5.17 deep-scrub starts 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[29608]: 5.17 deep-scrub ok 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:26:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:26:43.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: pgmap v311: 158 pgs: 158 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T11:26:43.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 5.17 deep-scrub starts 2024-03-24T11:26:43.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:42 smithi179 ceph-mon[37907]: 5.17 deep-scrub ok 2024-03-24T11:26:43.216 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:43.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:43.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:43.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:43.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:43.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:43.451 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:43.483 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:43.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:43.505 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:43.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:43.618 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:43.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:43.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:43.753 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:43.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:43.800 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:26:43.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:43.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:43.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:43.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:44.002 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total time run: 90.0059 2024-03-24T11:26:44.002 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total writes made: 128039 2024-03-24T11:26:44.002 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Write size: 65536 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object size: 65536 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Bandwidth (MB/sec): 88.9101 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Bandwidth: 31.9898 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max bandwidth (MB/sec): 223.312 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min bandwidth (MB/sec): 48.5625 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average IOPS: 1422 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev IOPS: 511.836 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max IOPS: 3573 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min IOPS: 777 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average Latency(s): 0.0112435 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Latency(s): 0.00864248 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max latency(s): 0.165789 2024-03-24T11:26:44.003 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min latency(s): 0.00127075 2024-03-24T11:26:44.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:44.008 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T11:26:44.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:44.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:44.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: 8.a scrub ok 2024-03-24T11:26:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: 6.4 deep-scrub starts 2024-03-24T11:26:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: 6.4 deep-scrub ok 2024-03-24T11:26:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:26:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:26:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:26:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:26:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:43 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: 8.a scrub ok 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: 6.4 deep-scrub starts 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: 6.4 deep-scrub ok 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: 8.a scrub ok 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: 6.4 deep-scrub starts 2024-03-24T11:26:44.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: 6.4 deep-scrub ok 2024-03-24T11:26:44.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:26:44.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:26:44.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:26:44.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:26:44.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:43 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:44.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.285 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:44.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:44.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.412 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:44.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:44.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:44.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:44.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:44.690 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:44.704 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:26:44.705 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:26:44.705 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T11:26:44.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T11:26:44.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:44.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:44.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:44.970 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:44.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:45.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:45.104 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:45.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:45.205 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:45.243 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:45.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:45.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:45.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:45 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:26:45.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:45 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:26:45.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:45 smithi181 ceph-mon[33998]: pgmap v312: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 893 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:26:45.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:45 smithi181 ceph-mon[33998]: 8.7 scrub ok 2024-03-24T11:26:45.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:45 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:26:45.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:45 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:26:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:45.370 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:45.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:26:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:26:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[29608]: pgmap v312: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 893 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:26:45.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[29608]: 8.7 scrub ok 2024-03-24T11:26:45.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:26:45.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:26:45.461 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:26:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:26:45.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:26:45.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:26:45.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[37907]: pgmap v312: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 893 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:26:45.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[37907]: 8.7 scrub ok 2024-03-24T11:26:45.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:26:45.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:45 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:26:45.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:45.506 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:45.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:45.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:45.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:45.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:45.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:45.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:45.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:45.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:45.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:45.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:46.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:46.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:46.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:46.090 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:26:46.090 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:26:46.090 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:26:46.090 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:26:46.090 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:26:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:46 smithi181 ceph-mon[33998]: 8.5 scrub starts 2024-03-24T11:26:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:46 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:26:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:46 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:26:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:46 smithi181 ceph-mon[33998]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:26:46.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:46.221 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:46.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:46.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:46.364 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:46.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[29608]: 8.5 scrub starts 2024-03-24T11:26:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:26:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:26:46.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[29608]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:26:46.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[37907]: 8.5 scrub starts 2024-03-24T11:26:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:26:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:26:46.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:46 smithi179 ceph-mon[37907]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T11:26:46.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:46.507 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:46.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:46.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:46.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:46.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:46.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:46.784 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:46.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:46.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:26:46.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:46.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:46.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:47.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:47.089 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:47.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:47.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:47.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:47.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:47.315 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:47.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: 8.9 scrub starts 2024-03-24T11:26:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: 8.2 scrub starts 2024-03-24T11:26:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: pgmap v314: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 921 B/s rd, 61 MiB/s wr, 978 op/s 2024-03-24T11:26:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: 3.b scrub starts 2024-03-24T11:26:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: 3.b scrub ok 2024-03-24T11:26:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:26:47.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:26:47.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:26:47.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:47 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:26:47.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:47.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: 8.9 scrub starts 2024-03-24T11:26:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: 8.2 scrub starts 2024-03-24T11:26:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: pgmap v314: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 921 B/s rd, 61 MiB/s wr, 978 op/s 2024-03-24T11:26:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: 3.b scrub starts 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: 3.b scrub ok 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: 8.9 scrub starts 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: 8.2 scrub starts 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: pgmap v314: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 921 B/s rd, 61 MiB/s wr, 978 op/s 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: 3.b scrub starts 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: 3.b scrub ok 2024-03-24T11:26:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:26:47.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:26:47.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:26:47.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:47 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:26:47.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:47.556 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:47.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:47.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:47.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:47.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:47.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:47.842 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:47.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:47.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:47.980 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:47.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:48.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:48.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:48.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:48.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:48.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:48.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:48.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:48.398 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:48.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:48.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:48 smithi181 ceph-mon[33998]: 8.5 scrub ok 2024-03-24T11:26:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:48 smithi181 ceph-mon[33998]: 3.12 deep-scrub starts 2024-03-24T11:26:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:48 smithi181 ceph-mon[33998]: 3.12 deep-scrub ok 2024-03-24T11:26:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:48 smithi181 ceph-mon[33998]: 8.d scrub starts 2024-03-24T11:26:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:48 smithi181 ceph-mon[33998]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:26:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:48.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[29608]: 8.5 scrub ok 2024-03-24T11:26:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[29608]: 3.12 deep-scrub starts 2024-03-24T11:26:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[29608]: 3.12 deep-scrub ok 2024-03-24T11:26:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[29608]: 8.d scrub starts 2024-03-24T11:26:48.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[29608]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[37907]: 8.5 scrub ok 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[37907]: 3.12 deep-scrub starts 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[37907]: 3.12 deep-scrub ok 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[37907]: 8.d scrub starts 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[37907]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:26:48.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:26:48.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:48.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:48.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:48.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:48.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:48.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:48.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:48.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:48.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:48.916 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:48.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:48.968 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:48.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:49.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:49.107 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:49.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:49.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:49.226 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26498 7.0 7:0d5c2bb1:::benchmark_data_smithi179_78267_object26497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 331.09621456500003, 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14803461000000001, 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:49.227 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26498 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125528+0000", 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.584e-06 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125569+0000", 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0945000000000002e-05 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125586+0000", 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7198999999999999e-05 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125596+0000", 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4919999999999999e-06 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125670+0000", 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4246999999999996e-05 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125813+0000", 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014268000000000001 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126002+0000", 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000189626 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126981+0000", 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00097871299999999998 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127606+0000", 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062507399999999999 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273447+0000", 2024-03-24T11:26:49.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14584118800000001 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273512+0000", 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4628999999999994e-05 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273561+0000", 2024-03-24T11:26:49.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9233000000000001e-05 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126626 7.0 7:0b9b7f0f:::benchmark_data_smithi179_78267_object7432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 244.645518055, 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10890327499999999, 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.234 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126626 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:26:49.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576224+0000", 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.172e-06 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576229+0000", 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5279999999999997e-06 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576240+0000", 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0963e-05 2024-03-24T11:26:49.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576248+0000", 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6589999999999996e-06 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576302+0000", 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3278e-05 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576422+0000", 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119988 2024-03-24T11:26:49.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576496+0000", 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4645000000000001e-05 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.579163+0000", 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002667123 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580732+0000", 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001568124 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685008+0000", 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10427672 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685071+0000", 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2853000000000003e-05 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685126+0000", 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5222000000000001e-05 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.241 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126640 7.0 7:05a202e2:::benchmark_data_smithi179_78267_object7446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 244.63056983000001, 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097235629000000004, 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126640 2024-03-24T11:26:49.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591173+0000", 2024-03-24T11:26:49.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.327e-06 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591177+0000", 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0589999999999996e-06 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591188+0000", 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0954e-05 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591194+0000", 2024-03-24T11:26:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3400000000000003e-06 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591252+0000", 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7930999999999998e-05 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591368+0000", 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115586 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591497+0000", 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129834 2024-03-24T11:26:49.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.593983+0000", 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024858359999999999 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.594318+0000", 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033431900000000002 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688305+0000", 2024-03-24T11:26:49.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093987494000000005 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688361+0000", 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6203999999999999e-05 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688407+0000", 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5745000000000003e-05 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:154480 7.0 7:09166c18:::benchmark_data_smithi179_78267_object35286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 214.12206604900001, 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099165692999999999, 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 154480 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:26:49.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099677+0000", 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.575e-06 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099684+0000", 2024-03-24T11:26:49.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6120000000000001e-06 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099697+0000", 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2541000000000001e-05 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099704+0000", 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5499999999999997e-06 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099976+0000", 2024-03-24T11:26:49.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000271122 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.100084+0000", 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010849900000000001 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.100211+0000", 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012710099999999999 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.103105+0000", 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028942939999999999 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106238+0000", 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003132953 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198758+0000", 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092520054000000004 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198797+0000", 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8912000000000003e-05 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198841+0000", 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3479999999999997e-05 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:216710 7.0 7:0ea60a8d:::benchmark_data_smithi179_78267_object97516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:19.012852+0000", 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 150.20888958500001, 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.129447284, 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 216710 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012852+0000", 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012852+0000", 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012853+0000", 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.114e-06 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012857+0000", 2024-03-24T11:26:49.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6360000000000002e-06 2024-03-24T11:26:49.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:49.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012868+0000", 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1141000000000001e-05 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012875+0000", 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2870000000000001e-06 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012923+0000", 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8678999999999999e-05 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013052+0000", 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012856200000000001 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013130+0000", 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7836999999999994e-05 2024-03-24T11:26:49.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017366+0000", 2024-03-24T11:26:49.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004235684 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017565+0000", 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019948200000000001 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142194+0000", 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.124628823 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142255+0000", 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0783999999999999e-05 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142299+0000", 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4255000000000002e-05 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:216711 7.0 7:01cdc540:::benchmark_data_smithi179_78267_object97517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 150.208722483, 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12939452500000001, 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 216711 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.264 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:26:49.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.37e-07 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013023+0000", 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0489999999999997e-06 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013034+0000", 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0847e-05 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013040+0000", 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1739999999999997e-06 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.015341+0000", 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023009079999999999 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.015431+0000", 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9692999999999997e-05 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.015511+0000", 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9919000000000005e-05 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017510+0000", 2024-03-24T11:26:49.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001999163 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017684+0000", 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017392800000000001 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142343+0000", 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12465865199999999 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142377+0000", 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3954000000000001e-05 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142413+0000", 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6501000000000002e-05 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:15737 8.8 8:1331dead:::benchmark_data_smithi179_105658_object15736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 89.719340591999995, 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.108844888, 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 15737 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.271 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2699999999999999e-07 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502439+0000", 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8068000000000002e-05 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502447+0000", 2024-03-24T11:26:49.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2619999999999998e-06 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502454+0000", 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2859999999999996e-06 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502522+0000", 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8664000000000005e-05 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502628+0000", 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105179 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502779+0000", 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151512 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.504075+0000", 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00129641 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.504167+0000", 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1201999999999999e-05 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.611160+0000", 2024-03-24T11:26:49.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10699328299999999 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.611208+0000", 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7743e-05 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.611245+0000", 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7752e-05 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.277 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:30245 8.8 8:11892159:::benchmark_data_smithi179_105658_object30244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:25:26.399919+0000", 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 82.821822439000002, 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.091839620999999996, 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.278 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 30245 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.399919+0000", 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.399919+0000", 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.399920+0000", 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8599999999999997e-07 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.399980+0000", 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0659000000000003e-05 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.399992+0000", 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1897e-05 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.400002+0000", 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7079999999999992e-06 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.400052+0000", 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0194000000000001e-05 2024-03-24T11:26:49.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.400134+0000", 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1903999999999994e-05 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.400220+0000", 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6106000000000005e-05 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.402201+0000", 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019807000000000002 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.402212+0000", 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1103e-05 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.491632+0000", 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.089419955999999995 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.491691+0000", 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9709e-05 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:26.491758+0000", 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6898999999999998e-05 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.284 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:75888 8.8 8:14095f93:::benchmark_data_smithi179_105658_object75887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:25:56.943427+0000", 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 52.278314602000002, 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082960987999999999, 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 75888 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.943427+0000", 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.943427+0000", 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.943427+0000", 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7700000000000004e-07 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.943457+0000", 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0201000000000001e-05 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.943465+0000", 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6020000000000002e-06 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.943471+0000", 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4200000000000004e-06 2024-03-24T11:26:49.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.947973+0000", 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045019459999999997 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.948101+0000", 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000127639 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.948246+0000", 2024-03-24T11:26:49.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014463599999999999 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.950484+0000", 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022382650000000001 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:56.950497+0000", 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3531e-05 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.026287+0000", 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.075789988000000003 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.026344+0000", 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6745e-05 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.026388+0000", 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3438e-05 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:76518 8.8 8:13c5a620:::benchmark_data_smithi179_105658_object76517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:25:57.491439+0000", 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.730302107, 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088530470999999999, 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.291 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 76518 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491439+0000", 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491439+0000", 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491440+0000", 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1699999999999997e-07 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491497+0000", 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7422e-05 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491512+0000", 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4912000000000001e-05 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491521+0000", 2024-03-24T11:26:49.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4600000000000003e-06 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491660+0000", 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013890800000000001 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491790+0000", 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013080200000000001 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.491945+0000", 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015472700000000001 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.494180+0000", 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002235112 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.494537+0000", 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00035693900000000002 2024-03-24T11:26:49.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.579891+0000", 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085353929999999995 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.579935+0000", 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3744e-05 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.579970+0000", 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4598000000000003e-05 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:76529 8.8 8:178ca411:::benchmark_data_smithi179_105658_object76528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:25:57.498965+0000", 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.722776226999997, 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.083201979999999995, 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.298 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 76529 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.498965+0000", 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.498965+0000", 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.498966+0000", 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8400000000000004e-07 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.499028+0000", 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2439000000000003e-05 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.499039+0000", 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0847e-05 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.499047+0000", 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8599999999999993e-06 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.499086+0000", 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8999999999999999e-05 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.499178+0000", 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2205000000000002e-05 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.499274+0000", 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6410999999999996e-05 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.501644+0000", 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023695779999999998 2024-03-24T11:26:49.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.501703+0000", 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9005000000000002e-05 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.582046+0000", 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.080342866999999998 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.582107+0000", 2024-03-24T11:26:49.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1214e-05 2024-03-24T11:26:49.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:57.582167+0000", 2024-03-24T11:26:49.306 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:49.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:49.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9870000000000001e-05 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:113237 8.d 8:b33d5dd6:::benchmark_data_smithi179_105658_object113236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.143843+0000", 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.077897913000001, 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.091849643999999994, 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.307 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113237 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.143843+0000", 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.143843+0000", 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.143846+0000", 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.328e-06 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.143881+0000", 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5652999999999999e-05 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.143895+0000", 2024-03-24T11:26:49.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3769e-05 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.143903+0000", 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1859999999999992e-06 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.143959+0000", 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5631999999999999e-05 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.144111+0000", 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000152254 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.144551+0000", 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044030099999999998 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.151014+0000", 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064626459999999998 2024-03-24T11:26:49.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.151054+0000", 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9509999999999999e-05 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235592+0000", 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084538931999999997 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235647+0000", 2024-03-24T11:26:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4617000000000002e-05 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235693+0000", 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5816000000000003e-05 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:113239 8.d 8:ba600ee7:::benchmark_data_smithi179_105658_object113238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.145108+0000", 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.076633534999999, 2024-03-24T11:26:49.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.091293699000000006, 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113239 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.145108+0000", 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.145108+0000", 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.145109+0000", 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9700000000000005e-07 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.145176+0000", 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7689999999999997e-05 2024-03-24T11:26:49.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.145187+0000", 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1045e-05 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.145195+0000", 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8550000000000002e-06 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.147861+0000", 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026656789999999998 2024-03-24T11:26:49.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.147954+0000", 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3447000000000002e-05 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148067+0000", 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112978 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154244+0000", 2024-03-24T11:26:49.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061764369999999999 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154266+0000", 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2464e-05 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.236317+0000", 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082050908000000006 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.236364+0000", 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7151000000000003e-05 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.236401+0000", 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7147999999999997e-05 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:113241 8.2 e175/150 8:41dd97b5:::benchmark_data_smithi179_105658_object113240:head v 175'6945, mlcod=175'6943)", 2024-03-24T11:26:49.319 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.148826+0000", 2024-03-24T11:26:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.072914926999999, 2024-03-24T11:26:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093705785999999999, 2024-03-24T11:26:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.320 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:26:49.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:26:49.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:26:49.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:26:49.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: pgmap v316: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 797 op/s 2024-03-24T11:26:49.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: 8.2 scrub ok 2024-03-24T11:26:49.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: 5.c deep-scrub starts 2024-03-24T11:26:49.321 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:49 smithi181 ceph-mon[33998]: 5.c deep-scrub ok 2024-03-24T11:26:49.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148826+0000", 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148826+0000", 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148827+0000", 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9299999999999997e-07 2024-03-24T11:26:49.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148847+0000", 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0044999999999999e-05 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148856+0000", 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7600000000000008e-06 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148861+0000", 2024-03-24T11:26:49.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.164e-06 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148911+0000", 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9960000000000003e-05 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148929+0000", 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8505e-05 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242481+0000", 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093551492 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242532+0000", 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1066999999999997e-05 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:113245 8.9 e175/150 8:9f18dbae:::benchmark_data_smithi179_105658_object113244:head v 175'6933, mlcod=175'6931)", 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.150347+0000", 2024-03-24T11:26:49.325 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.071394027, 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090922829999999996, 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150347+0000", 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150347+0000", 2024-03-24T11:26:49.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150348+0000", 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1500000000000004e-07 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150369+0000", 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1458e-05 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150381+0000", 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.153e-05 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150388+0000", 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2119999999999997e-06 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.151060+0000", 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067191600000000005 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.151078+0000", 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8459000000000001e-05 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.241224+0000", 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090145571999999993 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.241270+0000", 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5967999999999998e-05 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.329 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:113244 8.a e175/150 8:53878b67:::benchmark_data_smithi179_105658_object113243:head v 175'7000, mlcod=175'6998)", 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.150428+0000", 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.071313486000001, 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092321297999999996, 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150428+0000", 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150428+0000", 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150428+0000", 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5199999999999996e-07 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150485+0000", 2024-03-24T11:26:49.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6894000000000001e-05 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150495+0000", 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0152e-05 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150502+0000", 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3380000000000001e-06 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150546+0000", 2024-03-24T11:26:49.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4180000000000001e-05 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.150565+0000", 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8989999999999999e-05 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242702+0000", 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092137319999999995 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242749+0000", 2024-03-24T11:26:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6672000000000003e-05 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:113248 8.8 8:1495f646:::benchmark_data_smithi179_105658_object113247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.152833+0000", 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.068907670000002, 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.083145227000000002, 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:49.334 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113248 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.152833+0000", 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.152833+0000", 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.152835+0000", 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.88e-06 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.152879+0000", 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3878999999999998e-05 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.152893+0000", 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3779e-05 2024-03-24T11:26:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.152902+0000", 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6340000000000007e-06 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.152962+0000", 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0705999999999998e-05 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.153091+0000", 2024-03-24T11:26:49.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000128381 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.153233+0000", 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014190700000000001 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235776+0000", 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082543604000000007 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235823+0000", 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7045000000000001e-05 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235858+0000", 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4866999999999997e-05 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235938+0000", 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9596000000000002e-05 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235979+0000", 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0948999999999997e-05 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:113243 8.2 e175/150 8:4c9bdbcd:::benchmark_data_smithi179_105658_object113242:head v 175'6946, mlcod=175'6943)", 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.154300+0000", 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.067440946000001, 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088527368999999995, 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.340 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154300+0000", 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154300+0000", 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154301+0000", 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4e-07 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154326+0000", 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5398e-05 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154335+0000", 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7700000000000007e-06 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154341+0000", 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2020000000000001e-06 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154396+0000", 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5049000000000001e-05 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154415+0000", 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8868999999999998e-05 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242754+0000", 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088339002999999999 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242828+0000", 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3238000000000006e-05 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:113247 8.9 e175/150 8:9013d74a:::benchmark_data_smithi179_105658_object113246:head v 175'6934, mlcod=175'6931)", 2024-03-24T11:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.154703+0000", 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.067037717000002, 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.086600600999999999, 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154703+0000", 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154703+0000", 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154704+0000", 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9100000000000003e-07 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.154734+0000", 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0351000000000001e-05 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235337+0000", 2024-03-24T11:26:49.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.080602073999999996 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235348+0000", 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1219e-05 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235398+0000", 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0708999999999998e-05 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235424+0000", 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5442000000000001e-05 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.241279+0000", 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005854672 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.241304+0000", 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5443e-05 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.24959.0:113249 8.2 e175/150 8:48db2c3e:::benchmark_data_smithi179_105658_object113248:head v 175'6947, mlcod=175'6943)", 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.156525+0000", 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.065215977000001, 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.086341487999999994, 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156525+0000", 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156525+0000", 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156526+0000", 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3200000000000004e-07 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156556+0000", 2024-03-24T11:26:49.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9604999999999999e-05 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235331+0000", 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078775030999999995 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.235343+0000", 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2439000000000001e-05 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:49.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.238073+0000", 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002729853 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.238092+0000", 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9335e-05 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242831+0000", 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047386939999999999 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:26:49.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242867+0000", 2024-03-24T11:26:49.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5698999999999998e-05 2024-03-24T11:26:49.353 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.353 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.353 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.353 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:49.353 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:49.353 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: pgmap v316: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 797 op/s 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: 8.2 scrub ok 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: 5.c deep-scrub starts 2024-03-24T11:26:49.373 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[37907]: 5.c deep-scrub ok 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: pgmap v316: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 797 op/s 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: 8.2 scrub ok 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: 5.c deep-scrub starts 2024-03-24T11:26:49.374 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:49 smithi179 ceph-mon[29608]: 5.c deep-scrub ok 2024-03-24T11:26:49.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:49.449 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:49.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:49.550 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:49.595 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:49.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:49.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:49.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:49.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:49.833 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:49.874 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:49.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:49.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:50.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:50.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:50.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:50.164 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:26:50.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:50.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:50.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:50.421 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: 8.9 scrub ok 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: 6.7 deep-scrub starts 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: 6.7 deep-scrub ok 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4170073304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:26:50.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: 8.9 scrub ok 2024-03-24T11:26:50.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:26:50.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:26:50.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: 6.7 deep-scrub starts 2024-03-24T11:26:50.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: 6.7 deep-scrub ok 2024-03-24T11:26:50.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4170073304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:26:50.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:26:50.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:50 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:26:50.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:50.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:26:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.571 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:50.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: 8.9 scrub ok 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: 6.7 deep-scrub starts 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: 6.7 deep-scrub ok 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4170073304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:26:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:50 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:26:50.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.704 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:50.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:50.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:50.850 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:50.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:50.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:50.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:51.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:51.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:51.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:51.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:51.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:51.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:51.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:51.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:26:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:51.389 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:51.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:51.422 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: 8.d scrub ok 2024-03-24T11:26:51.423 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:26:51.423 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:26:51.423 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: 8.8 scrub starts 2024-03-24T11:26:51.423 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:51.423 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4170073304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:26:51.423 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:26:51.423 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:51 smithi181 ceph-mon[33998]: pgmap v318: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1022 B/s rd, 4.7 MiB/s wr, 845 op/s 2024-03-24T11:26:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: 8.d scrub ok 2024-03-24T11:26:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:26:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:26:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: 8.8 scrub starts 2024-03-24T11:26:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4170073304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[29608]: pgmap v318: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1022 B/s rd, 4.7 MiB/s wr, 845 op/s 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: 8.d scrub ok 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: 8.8 scrub starts 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4170073304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T11:26:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:51 smithi179 ceph-mon[37907]: pgmap v318: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1022 B/s rd, 4.7 MiB/s wr, 845 op/s 2024-03-24T11:26:51.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:51.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:51.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:51.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:51.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:51.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:51.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:51.754 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:51.794 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:51.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:51.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:51.935 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:51.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:52.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:52.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:52.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:52.172 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:52.214 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:52.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:52.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:52.363 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:52.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:52.454 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:52.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:52.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:52.532 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:52.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:52.633 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:52.674 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:52.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:26:52.774 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:26:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: "epoch": 179, 2024-03-24T11:26:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: "fsid": "6430538c-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:26:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: "created": "2024-03-24T11:13:18.639983+0000", 2024-03-24T11:26:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: "modified": "2024-03-24T11:26:50.142277+0000", 2024-03-24T11:26:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: "last_up_change": "2024-03-24T11:18:40.079955+0000", 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "last_in_change": "2024-03-24T11:18:23.090476+0000", 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_num": 5801984, 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_set": [ 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "noscrub", 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "pglog_hardlimit", 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "purged_snapdirs", 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "recovery_deletes", 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "sortbitwise" 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_version": 18, 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T11:26:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_snapshot": "", 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_max": 8, 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "max_osd": 8, 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "require_min_compat_client": "luminous", 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "min_compat_client": "luminous", 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "require_osd_release": "squid", 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "allow_crimson": false, 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "pools": [ 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 1, 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": ".mgr", 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:16:40.018600+0000", 2024-03-24T11:26:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 11, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 11, 2024-03-24T11:26:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 11, 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 11, 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 11, 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "138", 2024-03-24T11:26:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "136", 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:26:52.779 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:26:52.780 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "options": { 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_max": 32, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_min": 1 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:26:52.781 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr": {} 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 2.1800000667572021, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 2.1800000667572021, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 2, 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "datapool", 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:51.329095+0000", 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 8193, 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:26:52.783 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 3, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 3, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 3, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 3, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 3, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:26:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "69", 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 2, 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 69, 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:26:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:26:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:26:52.787 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd": {} 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 2.6500000953674316, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 2.6500000953674316, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:26:52.788 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 3, 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": ".rgw.root", 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:52.018113+0000", 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:26:52.789 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:26:52.790 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "67", 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:26:52.791 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:26:52.792 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.793 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:26:52.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.5, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.5, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 1.5, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 1.5, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.795 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 4, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "default.rgw.log", 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:53.978924+0000", 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:26:52.796 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.797 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "69", 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:26:52.798 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:26:52.799 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:26:52.800 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 2.25, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 2.25, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 2.25, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 2.25, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:26:52.801 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 5, 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "default.rgw.control", 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:55.925755+0000", 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:26:52.802 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:26:52.803 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "71", 2024-03-24T11:26:52.804 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:26:52.805 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:26:52.806 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:26:52.807 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.25, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.25, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 1.25, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 1.25, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.808 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.809 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 6, 2024-03-24T11:26:52.809 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "default.rgw.meta", 2024-03-24T11:26:52.809 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:57.991833+0000", 2024-03-24T11:26:52.809 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:26:52.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:52.810 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:26:52.810 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:26:52.811 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "73", 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:26:52.812 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:26:52.813 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:26:52.814 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "options": { 2024-03-24T11:26:52.815 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_bias": 4 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.75, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.75, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 1.75, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 1.75, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:26:52.816 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 8, 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "unique_pool_1", 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:25:01.155264+0000", 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:26:52.817 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 16, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 16, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 16, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 16, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 16, 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:26:52.818 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "156", 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:26:52.819 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:26:52.820 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:26:52.821 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:26:52.822 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "rados": {} 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 2, 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 2, 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1, 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 2, 2024-03-24T11:26:52.823 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 2, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 1, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 1 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "osds": [ 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 0, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b", 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.824 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 8, 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 150, 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6802", 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.825 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6803", 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6804", 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.826 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6805", 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6808", 2024-03-24T11:26:52.827 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6809", 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.828 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6806", 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6807", 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6803/1725503377", 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6805/1725503377", 2024-03-24T11:26:52.829 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6809/1725503377", 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6807/1725503377", 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 1, 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667", 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.830 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 13, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 150, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6810", 2024-03-24T11:26:52.831 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6811", 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.832 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6812", 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6813", 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.833 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6816", 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6817", 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.834 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6814", 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.835 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6815", 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6811/2564847275", 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6813/2564847275", 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6817/2564847275", 2024-03-24T11:26:52.836 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6815/2564847275", 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 2, 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc", 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.837 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 18, 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 92, 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6818", 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.838 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6819", 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.839 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6820", 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6821", 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.840 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6824", 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6825", 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.841 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6822", 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6823", 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:26:52.842 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6819/3297713842", 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6821/3297713842", 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6825/3297713842", 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6823/3297713842", 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 3, 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52", 2024-03-24T11:26:52.843 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 26, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 150, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.844 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6826", 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6827", 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.845 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6828", 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.846 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6829", 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6832", 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.847 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6833", 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6830", 2024-03-24T11:26:52.848 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6831", 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6827/944142471", 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6829/944142471", 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6833/944142471", 2024-03-24T11:26:52.849 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6831/944142471", 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 4, 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176", 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.850 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 31, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 150, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.851 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6800", 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6801", 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.852 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6802", 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6803", 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.853 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6806", 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.854 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6807", 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.855 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6804", 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6805", 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6801/3231660415", 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6803/3231660415", 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6807/3231660415", 2024-03-24T11:26:52.856 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6805/3231660415", 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 5, 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646", 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.857 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 41, 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 150, 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6808", 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.858 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6809", 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6810", 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.859 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6811", 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6814", 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.860 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6815", 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6812", 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.861 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6813", 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6809/775575782", 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6811/775575782", 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6815/775575782", 2024-03-24T11:26:52.862 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6813/775575782", 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 6, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793", 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.863 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 50, 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 150, 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6816", 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6817", 2024-03-24T11:26:52.864 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.865 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6818", 2024-03-24T11:26:52.866 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.866 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.866 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.866 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.866 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6819", 2024-03-24T11:26:52.866 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.866 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6822", 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6823", 2024-03-24T11:26:52.867 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6820", 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6821", 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:26:52.868 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6817/2357509374", 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6819/2357509374", 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6823/2357509374", 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6821/2357509374", 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 7, 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00", 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:26:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 59, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 141, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6824", 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6825", 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:26:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6826", 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6827", 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:26:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6830", 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6831", 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6828", 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6829", 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6825/732039117", 2024-03-24T11:26:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6827/732039117", 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6831/732039117", 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6829/732039117", 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_xinfo": [ 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 0, 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:25:44.718587+0000", 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 1, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:26:44.899101+0000", 2024-03-24T11:26:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 2, 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:26:09.963932+0000", 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 3, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:26:46.407825+0000", 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 4, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:26:09.176139+0000", 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 5, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:25:39.140997+0000", 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 6, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:26:00.995311+0000", 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 7, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:26:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:26:17.126422+0000", 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_upmap": [], 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_upmap_items": [ 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "pgid": "4.7", 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "mappings": [ 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "from": 1, 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "to": 2 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "pgid": "4.d", 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "mappings": [ 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "from": 1, 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "to": 5 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "pgid": "4.18", 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "mappings": [ 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "from": 1, 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "to": 2 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_upmap_primaries": [], 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_temp": [], 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_temp": [], 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "blocklist": { 2024-03-24T11:26:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/1519226585": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6800/2865046672": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2472348284": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2047439415": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6801/3169437": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6800/3169437": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6801/243537818": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/3185072343": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/3344305045": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2748363950": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6800/243537818": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:26:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2066930919": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2108875546": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6801/2865046672": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/4087802319": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/883221702": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/4255117010": "2024-03-25T11:19:31.961858+0000" 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "range_blocklist": {}, 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profiles": { 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "default": { 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "crush-failure-domain": "osd", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "k": "2", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "m": "1", 2024-03-24T11:26:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "plugin": "jerasure", 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "technique": "reed_sol_van" 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps_queue": [], 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "new_removed_snaps": [], 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "new_purged_snaps": [], 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_node_flags": {}, 2024-03-24T11:26:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "device_class_flags": {}, 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_mode": { 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_mode_enabled": false, 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_bucket_count": 0, 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: "degraded_stretch_mode": 0, 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: "recovering_stretch_mode": 0, 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_mode_bucket": 0 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:26:52.888 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:26:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:52.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:52.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:52.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:53.030 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:53.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:53.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:53.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:53.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:53.269 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:53.293 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:53.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:53.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:53.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:53.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:53.468 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:53 smithi181 ceph-mon[33998]: pgmap v319: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 872 B/s rd, 4.0 MiB/s wr, 721 op/s 2024-03-24T11:26:53.468 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/890382412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:26:53.529 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:53.566 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:53.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:53.593 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:53 smithi179 ceph-mon[29608]: pgmap v319: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 872 B/s rd, 4.0 MiB/s wr, 721 op/s 2024-03-24T11:26:53.593 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/890382412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:26:53.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:53 smithi179 ceph-mon[37907]: pgmap v319: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 872 B/s rd, 4.0 MiB/s wr, 721 op/s 2024-03-24T11:26:53.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/890382412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:26:53.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:53.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:53.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:53.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:26:53.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:53.879 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:53.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:53.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:26:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:26:53.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:54.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:54.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:54.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:54.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:54.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:54.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:54.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:54.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:54.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:54.440 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:54.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:54.482 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:54.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:54.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:54.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:54.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:54.707 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:54.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:26:54.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:54.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:54.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:54.867 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:54.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:54.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:139970 8.5 8:a7b1998e:::benchmark_data_smithi179_105658_object11928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:54.958309+0000", 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.028248692999999998, 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.028284409, 2024-03-24T11:26:54.993 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 139970 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958309+0000", 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958309+0000", 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958310+0000", 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.51e-07 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:54.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958313+0000", 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9110000000000002e-06 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958321+0000", 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9279999999999993e-06 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958327+0000", 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9610000000000002e-06 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958385+0000", 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8485e-05 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958488+0000", 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010291899999999999 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 4,7", 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.958592+0000", 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010401299999999999 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.964091+0000", 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054994309999999999 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.967168+0000", 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003076358 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:139978 8.9 8:9a23f5f9:::benchmark_data_smithi179_105658_object11936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:54.965489+0000", 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.021068415, 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021206051, 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 139978 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:54.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.965489+0000", 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.965489+0000", 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.965490+0000", 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0400000000000005e-07 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.965495+0000", 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8160000000000001e-06 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.965508+0000", 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3002e-05 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.965517+0000", 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5890000000000003e-06 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.968988+0000", 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034713600000000002 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.969096+0000", 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107459 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.969474+0000", 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037816400000000001 2024-03-24T11:26:55.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.976288+0000", 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0068138410000000002 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.976524+0000", 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023603099999999999 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:139993 8.5 8:a7849a73:::benchmark_data_smithi179_105658_object11951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:54.978864+0000", 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0076940330000000003, 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0079364239999999992, 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 139993 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:55.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.978864+0000", 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.978864+0000", 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.978865+0000", 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3319999999999999e-06 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.978868+0000", 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9179999999999998e-06 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.978877+0000", 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4970000000000001e-06 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.978882+0000", 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7910000000000001e-06 2024-03-24T11:26:55.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.978957+0000", 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4789999999999994e-05 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.979082+0000", 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000124941 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 4,7", 2024-03-24T11:26:55.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.979201+0000", 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118616 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:139994 8.9 8:93510204:::benchmark_data_smithi179_105658_object11952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:54.981415+0000", 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0051433889999999999, 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054515290000000001, 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:55.008 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 139994 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.981415+0000", 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.981415+0000", 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.981416+0000", 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5820000000000001e-06 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.981423+0000", 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7179999999999999e-06 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.981443+0000", 2024-03-24T11:26:55.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0021999999999999e-05 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.981453+0000", 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0047000000000001e-05 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982664+0000", 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012112080000000001 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982787+0000", 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122967 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982907+0000", 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011948899999999999 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:139995 8.5 8:a1ce9319:::benchmark_data_smithi179_105658_object11953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:54.982055+0000", 2024-03-24T11:26:55.012 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0045026210000000001, 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048791859999999998, 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 139995 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982055+0000", 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982055+0000", 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982056+0000", 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2200000000000003e-07 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:55.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982061+0000", 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7169999999999999e-06 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982071+0000", 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8290000000000003e-06 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982078+0000", 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5939999999999996e-06 2024-03-24T11:26:55.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982173+0000", 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5018000000000004e-05 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982285+0000", 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011168299999999999 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 4,7", 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982393+0000", 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010757299999999999 2024-03-24T11:26:55.016 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.24959.0:139964 8.5 e179/150)", 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:54.982347+0000", 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0042110380000000003, 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00465812, 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:55.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982347+0000", 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982347+0000", 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982348+0000", 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2440000000000001e-06 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982353+0000", 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2510000000000002e-06 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982365+0000", 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1769e-05 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.982378+0000", 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2894000000000001e-05 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.019 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:139964 8.5 8:aa3f6adf:::benchmark_data_smithi179_105658_object11922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:54.954802+0000", 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.031756064000000001, 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.032256730999999997, 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 139964 2024-03-24T11:26:55.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.954802+0000", 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.954802+0000", 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.954803+0000", 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.375e-06 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.954807+0000", 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2910000000000001e-06 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.954818+0000", 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0356e-05 2024-03-24T11:26:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.954825+0000", 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.092e-06 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.955077+0000", 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025216700000000003 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.955201+0000", 2024-03-24T11:26:55.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012406799999999999 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 4,7", 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.955319+0000", 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118311 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.964053+0000", 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0087338030000000001 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:54.967159+0000", 2024-03-24T11:26:55.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031058079999999998 2024-03-24T11:26:55.025 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.025 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:26:55.025 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.025 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:26:55.025 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-03-24T11:26:55.025 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 7 2024-03-24T11:26:55.025 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:26:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:55.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:55.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:55.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:55.181 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:55.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:55.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:26:55.329 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:55.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:26:55.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:55.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:55 smithi179 ceph-mon[29608]: pgmap v320: 158 pgs: 158 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.5 MiB/s wr, 1.32k op/s 2024-03-24T11:26:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:55 smithi179 ceph-mon[37907]: pgmap v320: 158 pgs: 158 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.5 MiB/s wr, 1.32k op/s 2024-03-24T11:26:55.460 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:26:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:26:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:26:55.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:55.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:55.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:55.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:55.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:55.629 INFO:tasks.thrashosds.thrasher:j is {'epoch': 179, 'fsid': '6430538c-e9cf-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:13:18.639983+0000', 'modified': '2024-03-24T11:26:50.142277+0000', 'last_up_change': '2024-03-24T11:18:40.079955+0000', 'last_in_change': '2024-03-24T11:18:23.090476+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:16:40.018600+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '136', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:18:51.329095+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:18:52.018113+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:18:53.978924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:18:55.925755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:18:57.991833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-24T11:25:01.155264+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '14f300f0-7498-4ead-a87e-d9f38bdff21b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 1725503377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 1725503377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 1725503377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 1725503377}]}, 'public_addr': '172.21.15.179:6803/1725503377', 'cluster_addr': '172.21.15.179:6805/1725503377', 'heartbeat_back_addr': '172.21.15.179:6809/1725503377', 'heartbeat_front_addr': '172.21.15.179:6807/1725503377', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 2564847275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 2564847275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2564847275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 2564847275}]}, 'public_addr': '172.21.15.179:6811/2564847275', 'cluster_addr': '172.21.15.179:6813/2564847275', 'heartbeat_back_addr': '172.21.15.179:6817/2564847275', 'heartbeat_front_addr': '172.21.15.179:6815/2564847275', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '863b92b8-c68c-41a3-aed4-fd79e96d34dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 3297713842}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 3297713842}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 3297713842}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 3297713842}]}, 'public_addr': '172.21.15.179:6819/3297713842', 'cluster_addr': '172.21.15.179:6821/3297713842', 'heartbeat_back_addr': '172.21.15.179:6825/3297713842', 'heartbeat_front_addr': '172.21.15.179:6823/3297713842', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c3d6b564-195d-4771-9aae-93a9e2eaba52', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 944142471}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 944142471}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6832', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6833', 'nonce': 944142471}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 944142471}]}, 'public_addr': '172.21.15.179:6827/944142471', 'cluster_addr': '172.21.15.179:6829/944142471', 'heartbeat_back_addr': '172.21.15.179:6833/944142471', 'heartbeat_front_addr': '172.21.15.179:6831/944142471', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91bf5303-3a94-407d-aef1-e43bf0e16176', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6801', 'nonce': 3231660415}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6803', 'nonce': 3231660415}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6807', 'nonce': 3231660415}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6805', 'nonce': 3231660415}]}, 'public_addr': '172.21.15.181:6801/3231660415', 'cluster_addr': '172.21.15.181:6803/3231660415', 'heartbeat_back_addr': '172.21.15.181:6807/3231660415', 'heartbeat_front_addr': '172.21.15.181:6805/3231660415', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cba3f489-e8be-46f7-a281-e69a0ad08646', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6809', 'nonce': 775575782}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6811', 'nonce': 775575782}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6815', 'nonce': 775575782}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6813', 'nonce': 775575782}]}, 'public_addr': '172.21.15.181:6809/775575782', 'cluster_addr': '172.21.15.181:6811/775575782', 'heartbeat_back_addr': '172.21.15.181:6815/775575782', 'heartbeat_front_addr': '172.21.15.181:6813/775575782', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44a2c3a0-b704-4c10-a257-c54de47ee793', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6816', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6817', 'nonce': 2357509374}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6818', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6819', 'nonce': 2357509374}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6822', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6823', 'nonce': 2357509374}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6820', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6821', 'nonce': 2357509374}]}, 'public_addr': '172.21.15.181:6817/2357509374', 'cluster_addr': '172.21.15.181:6819/2357509374', 'heartbeat_back_addr': '172.21.15.181:6823/2357509374', 'heartbeat_front_addr': '172.21.15.181:6821/2357509374', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'df60c4b4-d2f2-4e1f-8e1f-3169c403de00', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 141, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6824', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6825', 'nonce': 732039117}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6826', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6827', 'nonce': 732039117}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6830', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6831', 'nonce': 732039117}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6828', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6829', 'nonce': 732039117}]}, 'public_addr': '172.21.15.181:6825/732039117', 'cluster_addr': '172.21.15.181:6827/732039117', 'heartbeat_back_addr': '172.21.15.181:6831/732039117', 'heartbeat_front_addr': '172.21.15.181:6829/732039117', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:25:44.718587+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:26:44.899101+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:26:09.963932+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:26:46.407825+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:26:09.176139+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:25:39.140997+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:26:00.995311+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:26:17.126422+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.179:0/1519226585': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:6800/2865046672': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/2472348284': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/2047439415': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:6801/3169437': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:6800/3169437': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:6801/243537818': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:0/3185072343': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/3344305045': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:0/2748363950': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:6800/243537818': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:0/2066930919': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:0/2108875546': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:6801/2865046672': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/4087802319': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/883221702': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:0/4255117010': '2024-03-25T11:19:31.961858+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T11:26:55.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:26:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:55 smithi181 ceph-mon[33998]: pgmap v320: 158 pgs: 158 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.5 MiB/s wr, 1.32k op/s 2024-03-24T11:26:55.706 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:55.766 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:55.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:55.867 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:55.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:55.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:56.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:56.037 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:56.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:56.138 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:56.185 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:56.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:56.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:56.289 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:56.294 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:26:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:56.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:56.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:56.351 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:26:56.400 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:26:56.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:56.440 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:26:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:56.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:56.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:56.480 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:26:56.518 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:26:56.559 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:26:56.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:56.597 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:56.597 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:26:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:56.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:56.636 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:26:56.642 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:26:56.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:26:56.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:56.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:56.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:56.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:56.879 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:56.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:56.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:57.010 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:26:57.015 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:57.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:57.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:57.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:57.290 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:57.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:57.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:57 smithi181 ceph-mon[33998]: pgmap v321: 158 pgs: 158 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 MiB/s wr, 1.17k op/s 2024-03-24T11:26:57.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1532678653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:57.428 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:57.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:26:57.556 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:57 smithi179 ceph-mon[29608]: pgmap v321: 158 pgs: 158 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 MiB/s wr, 1.17k op/s 2024-03-24T11:26:57.556 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1532678653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:57.556 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:57 smithi179 ceph-mon[37907]: pgmap v321: 158 pgs: 158 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 MiB/s wr, 1.17k op/s 2024-03-24T11:26:57.557 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1532678653' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:26:57.566 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:26:57.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:57.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:57.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:57.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:57.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:57.972 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:57.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:58.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:58.107 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:58.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:58.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:58.248 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:58.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:58.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:58.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:58.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:58.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:58.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:58.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:58.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:26:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:58.679 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:58.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:26:58.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:58.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:58.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:58.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:26:58.960 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:26:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:58.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:26:59.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:59.098 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:26:59.098 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:26:59.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:59.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:59.205 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:26:59.239 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:59.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:26:59.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:59.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:59.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:59.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:26:59 smithi179 ceph-mon[29608]: pgmap v322: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1022 B/s rd, 1.6 MiB/s wr, 1.08k op/s 2024-03-24T11:26:59.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:26:59 smithi179 ceph-mon[37907]: pgmap v322: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1022 B/s rd, 1.6 MiB/s wr, 1.08k op/s 2024-03-24T11:26:59.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:59.491 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:26:59 smithi181 ceph-mon[33998]: pgmap v322: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1022 B/s rd, 1.6 MiB/s wr, 1.08k op/s 2024-03-24T11:26:59.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:59.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:26:59.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:26:59.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:59.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:26:59.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:26:59.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:59.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:26:59.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T11:26:59.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:26:59.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:26:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:26:59.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:00.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:00.113 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:00.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:00.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:00.252 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:00.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:00.278 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:27:00.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:00.400 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:00.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:00.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:27:00.501 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:00.560 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:00.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:00.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:00.693 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:00.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:00.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:00.838 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:00.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:00.939 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:01.005 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:01.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:01.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:01.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:01.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:01.365 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.414 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:01.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:01.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:01 smithi179 ceph-mon[29608]: pgmap v323: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:27:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:01 smithi179 ceph-mon[37907]: pgmap v323: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:27:01.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:01.519 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.545 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:01.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:01.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:01.653 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:27:01.656 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:01 smithi181 ceph-mon[33998]: pgmap v323: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:27:01.656 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:27:01.674 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:01.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.813 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:01.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:01.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:01.950 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:01.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:02.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:02.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:02.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:02.185 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:02.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:02.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:02.273 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:27:02.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:02.337 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:02 smithi181 ceph-mon[33998]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:27:02.337 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:02 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/258307597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:02.337 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:02 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:02.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:02.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:02.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:02.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:02.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:02.579 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:02 smithi179 ceph-mon[29608]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:27:02.580 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:02 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/258307597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:02.580 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:02 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:02.580 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:02 smithi179 ceph-mon[37907]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:27:02.580 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:02 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/258307597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:02.580 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:02 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:02.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:02.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:02.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:02.669 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-03-24T11:27:00.168428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7325882742,"num_objects":111974,"num_object_clones":0,"num_object_copies":335922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111974,"num_whiteouts":0,"num_read":1199,"num_read_kb":951,"num_write":144740,"num_write_kb":8194542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82544,"ondisk_log_size":82544,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23384296,"kb_used_data":21581288,"kb_used_omap":12,"kb_used_meta":1802803,"kb_avail":726576920,"statfs":{"total":767960285184,"available":744014766080,"internally_reserved":0,"allocated":22099238912,"data_stored":22096642242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1846070833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-762707968,"num_objects":-11638,"num_object_clones":0,"num_object_copies":-34914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11638,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.021635"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.166001+0000","last_change":"2024-03-24T11:26:32.919289+0000","last_active":"2024-03-24T11:26:50.166001+0000","last_peered":"2024-03-24T11:26:50.166001+0000","last_clean":"2024-03-24T11:26:50.166001+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:26:50.166001+0000","last_undegraded":"2024-03-24T11:26:50.166001+0000","last_fullsized":"2024-03-24T11:26:50.166001+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.919158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.919158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:46.842537+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.176477+0000","last_change":"2024-03-24T11:25:40.884239+0000","last_active":"2024-03-24T11:26:50.176477+0000","last_peered":"2024-03-24T11:26:50.176477+0000","last_clean":"2024-03-24T11:26:50.176477+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:26:50.176477+0000","last_undegraded":"2024-03-24T11:26:50.176477+0000","last_fullsized":"2024-03-24T11:26:50.176477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:40.884119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.884119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.178483+0000","last_change":"2024-03-24T11:25:36.241293+0000","last_active":"2024-03-24T11:26:50.178483+0000","last_peered":"2024-03-24T11:26:50.178483+0000","last_clean":"2024-03-24T11:26:50.178483+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:26:50.178483+0000","last_undegraded":"2024-03-24T11:26:50.178483+0000","last_fullsized":"2024-03-24T11:26:50.178483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:36.241166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.241166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:57.650891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":300,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164960+0000","last_change":"2024-03-24T11:26:40.754563+0000","last_active":"2024-03-24T11:26:50.164960+0000","last_peered":"2024-03-24T11:26:50.164960+0000","last_clean":"2024-03-24T11:26:50.164960+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:26:50.164960+0000","last_undegraded":"2024-03-24T11:26:50.164960+0000","last_fullsized":"2024-03-24T11:26:50.164960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:26:40.754425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:40.754425+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:40.933020+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":244,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157777+0000","last_change":"2024-03-24T11:25:38.404738+0000","last_active":"2024-03-24T11:26:50.157777+0000","last_peered":"2024-03-24T11:26:50.157777+0000","last_clean":"2024-03-24T11:26:50.157777+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:26:50.157777+0000","last_undegraded":"2024-03-24T11:26:50.157777+0000","last_fullsized":"2024-03-24T11:26:50.157777+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:38.404511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.404511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:50.603293+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164871+0000","last_change":"2024-03-24T11:25:39.063707+0000","last_active":"2024-03-24T11:26:50.164871+0000","last_peered":"2024-03-24T11:26:50.164871+0000","last_clean":"2024-03-24T11:26:50.164871+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:26:50.164871+0000","last_undegraded":"2024-03-24T11:26:50.164871+0000","last_fullsized":"2024-03-24T11:26:50.164871+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:39.063590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.063590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":257,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157843+0000","last_change":"2024-03-24T11:23:44.804746+0000","last_active":"2024-03-24T11:26:50.157843+0000","last_peered":"2024-03-24T11:26:50.157843+0000","last_clean":"2024-03-24T11:26:50.157843+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:26:50.157843+0000","last_undegraded":"2024-03-24T11:26:50.157843+0000","last_fullsized":"2024-03-24T11:26:50.157843+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:44.804631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:44.804631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:03.386820+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.158105+0000","last_change":"2024-03-24T11:26:29.288367+0000","last_active":"2024-03-24T11:26:50.158105+0000","last_peered":"2024-03-24T11:26:50.158105+0000","last_clean":"2024-03-24T11:26:50.158105+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:26:50.158105+0000","last_undegraded":"2024-03-24T11:26:50.158105+0000","last_fullsized":"2024-03-24T11:26:50.158105+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:26:29.288223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.288223+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:31.719337+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.169913+0000","last_change":"2024-03-24T11:24:48.401067+0000","last_active":"2024-03-24T11:26:50.169913+0000","last_peered":"2024-03-24T11:26:50.169913+0000","last_clean":"2024-03-24T11:26:50.169913+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:26:50.169913+0000","last_undegraded":"2024-03-24T11:26:50.169913+0000","last_fullsized":"2024-03-24T11:26:50.169913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:24:48.400935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:24:48.400935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.148546+0000","last_change":"2024-03-24T11:25:32.085917+0000","last_active":"2024-03-24T11:26:50.148546+0000","last_peered":"2024-03-24T11:26:50.148546+0000","last_clean":"2024-03-24T11:26:50.148546+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:26:50.148546+0000","last_undegraded":"2024-03-24T11:26:50.148546+0000","last_fullsized":"2024-03-24T11:26:50.148546+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:32.085822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:32.085822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149276+0000","last_change":"2024-03-24T11:25:46.067948+0000","last_active":"2024-03-24T11:26:50.149276+0000","last_peered":"2024-03-24T11:26:50.149276+0000","last_clean":"2024-03-24T11:26:50.149276+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:26:50.149276+0000","last_undegraded":"2024-03-24T11:26:50.149276+0000","last_fullsized":"2024-03-24T11:26:50.149276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:25:46.067824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:12.004038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":276,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.166096+0000","last_change":"2024-03-24T11:25:29.099118+0000","last_active":"2024-03-24T11:26:50.166096+0000","last_peered":"2024-03-24T11:26:50.166096+0000","last_clean":"2024-03-24T11:26:50.166096+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:26:50.166096+0000","last_undegraded":"2024-03-24T11:26:50.166096+0000","last_fullsized":"2024-03-24T11:26:50.166096+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:25:29.098999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:29.098999+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:53.207645+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.148430+0000","last_change":"2024-03-24T11:26:42.053183+0000","last_active":"2024-03-24T11:26:50.148430+0000","last_peered":"2024-03-24T11:26:50.148430+0000","last_clean":"2024-03-24T11:26:50.148430+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:26:50.148430+0000","last_undegraded":"2024-03-24T11:26:50.148430+0000","last_fullsized":"2024-03-24T11:26:50.148430+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.052777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.052777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:45.628691+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153409+0000","last_change":"2024-03-24T11:25:31.258073+0000","last_active":"2024-03-24T11:26:50.153409+0000","last_peered":"2024-03-24T11:26:50.153409+0000","last_clean":"2024-03-24T11:26:50.153409+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:26:50.153409+0000","last_undegraded":"2024-03-24T11:26:50.153409+0000","last_fullsized":"2024-03-24T11:26:50.153409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:31.257974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:25:31.257974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:54.596419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.148891+0000","last_change":"2024-03-24T11:26:43.090579+0000","last_active":"2024-03-24T11:26:50.148891+0000","last_peered":"2024-03-24T11:26:50.148891+0000","last_clean":"2024-03-24T11:26:50.148891+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:26:50.148891+0000","last_undegraded":"2024-03-24T11:26:50.148891+0000","last_fullsized":"2024-03-24T11:26:50.148891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:43.090367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.090367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:06.561184+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":278,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165059+0000","last_change":"2024-03-24T11:26:47.926021+0000","last_active":"2024-03-24T11:26:50.165059+0000","last_peered":"2024-03-24T11:26:50.165059+0000","last_clean":"2024-03-24T11:26:50.165059+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:26:50.165059+0000","last_undegraded":"2024-03-24T11:26:50.165059+0000","last_fullsized":"2024-03-24T11:26:50.165059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-24T11:26:47.925773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.925773+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:56.281392+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165540+0000","last_change":"2024-03-24T11:26:27.996936+0000","last_active":"2024-03-24T11:26:50.165540+0000","last_peered":"2024-03-24T11:26:50.165540+0000","last_clean":"2024-03-24T11:26:50.165540+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:26:50.165540+0000","last_undegraded":"2024-03-24T11:26:50.165540+0000","last_fullsized":"2024-03-24T11:26:50.165540+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:27.996815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.996815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:50.789628+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164455+0000","last_change":"2024-03-24T11:25:27.005214+0000","last_active":"2024-03-24T11:26:50.164455+0000","last_peered":"2024-03-24T11:26:50.164455+0000","last_clean":"2024-03-24T11:26:50.164455+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:26:50.164455+0000","last_undegraded":"2024-03-24T11:26:50.164455+0000","last_fullsized":"2024-03-24T11:26:50.164455+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:27.005093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:25:27.005093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:51.081720+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.177042+0000","last_change":"2024-03-24T11:26:32.451559+0000","last_active":"2024-03-24T11:26:50.177042+0000","last_peered":"2024-03-24T11:26:50.177042+0000","last_clean":"2024-03-24T11:26:50.177042+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:26:50.177042+0000","last_undegraded":"2024-03-24T11:26:50.177042+0000","last_fullsized":"2024-03-24T11:26:50.177042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.451446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.451446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:39.602821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":293,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.156329+0000","last_change":"2024-03-24T11:25:39.721869+0000","last_active":"2024-03-24T11:26:50.156329+0000","last_peered":"2024-03-24T11:26:50.156329+0000","last_clean":"2024-03-24T11:26:50.156329+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:26:50.156329+0000","last_undegraded":"2024-03-24T11:26:50.156329+0000","last_fullsized":"2024-03-24T11:26:50.156329+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:25:39.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.721719+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:04.020389+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":244,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.160457+0000","last_change":"2024-03-24T11:24:26.157701+0000","last_active":"2024-03-24T11:26:50.160457+0000","last_peered":"2024-03-24T11:26:50.160457+0000","last_clean":"2024-03-24T11:26:50.160457+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:26:50.160457+0000","last_undegraded":"2024-03-24T11:26:50.160457+0000","last_fullsized":"2024-03-24T11:26:50.160457+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:24:26.157583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:24:26.157583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165786+0000","last_change":"2024-03-24T11:26:26.941914+0000","last_active":"2024-03-24T11:26:50.165786+0000","last_peered":"2024-03-24T11:26:50.165786+0000","last_clean":"2024-03-24T11:26:50.165786+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:26:50.165786+0000","last_undegraded":"2024-03-24T11:26:50.165786+0000","last_fullsized":"2024-03-24T11:26:50.165786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.941809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.941809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:33.817485+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.166141+0000","last_change":"2024-03-24T11:25:30.083197+0000","last_active":"2024-03-24T11:26:50.166141+0000","last_peered":"2024-03-24T11:26:50.166141+0000","last_clean":"2024-03-24T11:26:50.166141+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:26:50.166141+0000","last_undegraded":"2024-03-24T11:26:50.166141+0000","last_fullsized":"2024-03-24T11:26:50.166141+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:30.082634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:30.082634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":275,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157389+0000","last_change":"2024-03-24T11:25:39.305828+0000","last_active":"2024-03-24T11:26:50.157389+0000","last_peered":"2024-03-24T11:26:50.157389+0000","last_clean":"2024-03-24T11:26:50.157389+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:26:50.157389+0000","last_undegraded":"2024-03-24T11:26:50.157389+0000","last_fullsized":"2024-03-24T11:26:50.157389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:25:39.305712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.305712+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:56.948340+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.154161+0000","last_change":"2024-03-24T11:24:48.431549+0000","last_active":"2024-03-24T11:26:50.154161+0000","last_peered":"2024-03-24T11:26:50.154161+0000","last_clean":"2024-03-24T11:26:50.154161+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:26:50.154161+0000","last_undegraded":"2024-03-24T11:26:50.154161+0000","last_fullsized":"2024-03-24T11:26:50.154161+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:24:48.431457+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_clean_scrub_stamp":"2024-03-24T11:24:48.431457+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":376,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.150613+0000","last_change":"2024-03-24T11:25:18.874358+0000","last_active":"2024-03-24T11:26:50.150613+0000","last_peered":"2024-03-24T11:26:50.150613+0000","last_clean":"2024-03-24T11:26:50.150613+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:26:50.150613+0000","last_undegraded":"2024-03-24T11:26:50.150613+0000","last_fullsized":"2024-03-24T11:26:50.150613+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:25:18.874179+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:23:53.129429+0000","last_clean_scrub_stamp":"2024-03-24T11:25:18.874179+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":281,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.176555+0000","last_change":"2024-03-24T11:26:26.669460+0000","last_active":"2024-03-24T11:26:50.176555+0000","last_peered":"2024-03-24T11:26:50.176555+0000","last_clean":"2024-03-24T11:26:50.176555+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:26:50.176555+0000","last_undegraded":"2024-03-24T11:26:50.176555+0000","last_fullsized":"2024-03-24T11:26:50.176555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:26:26.669216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.669216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:47.472918+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.181662+0000","last_change":"2024-03-24T11:25:35.265042+0000","last_active":"2024-03-24T11:26:50.181662+0000","last_peered":"2024-03-24T11:26:50.181662+0000","last_clean":"2024-03-24T11:26:50.181662+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:26:50.181662+0000","last_undegraded":"2024-03-24T11:26:50.181662+0000","last_fullsized":"2024-03-24T11:26:50.181662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:25:35.264757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.264757+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:02.444704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165229+0000","last_change":"2024-03-24T11:25:46.053120+0000","last_active":"2024-03-24T11:26:50.165229+0000","last_peered":"2024-03-24T11:26:50.165229+0000","last_clean":"2024-03-24T11:26:50.165229+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:26:50.165229+0000","last_undegraded":"2024-03-24T11:26:50.165229+0000","last_fullsized":"2024-03-24T11:26:50.165229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:46.052975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:58.788469+0000","last_clean_scrub_stamp":"2024-03-24T11:25:46.052975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:54.743133+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.181909+0000","last_change":"2024-03-24T11:26:31.491413+0000","last_active":"2024-03-24T11:26:50.181909+0000","last_peered":"2024-03-24T11:26:50.181909+0000","last_clean":"2024-03-24T11:26:50.181909+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:26:50.181909+0000","last_undegraded":"2024-03-24T11:26:50.181909+0000","last_fullsized":"2024-03-24T11:26:50.181909+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:31.491292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:31.491292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.469946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164052+0000","last_change":"2024-03-24T11:26:26.240418+0000","last_active":"2024-03-24T11:26:50.164052+0000","last_peered":"2024-03-24T11:26:50.164052+0000","last_clean":"2024-03-24T11:26:50.164052+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:26:50.164052+0000","last_undegraded":"2024-03-24T11:26:50.164052+0000","last_fullsized":"2024-03-24T11:26:50.164052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.240302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.240302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:38.755012+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":300,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.150482+0000","last_change":"2024-03-24T11:25:45.772885+0000","last_active":"2024-03-24T11:26:50.150482+0000","last_peered":"2024-03-24T11:26:50.150482+0000","last_clean":"2024-03-24T11:26:50.150482+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:26:50.150482+0000","last_undegraded":"2024-03-24T11:26:50.150482+0000","last_fullsized":"2024-03-24T11:26:50.150482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:25:45.772734+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:25:45.772734+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:05.011463+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":255,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153319+0000","last_change":"2024-03-24T11:25:22.230485+0000","last_active":"2024-03-24T11:26:50.153319+0000","last_peered":"2024-03-24T11:26:50.153319+0000","last_clean":"2024-03-24T11:26:50.153319+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:26:50.153319+0000","last_undegraded":"2024-03-24T11:26:50.153319+0000","last_fullsized":"2024-03-24T11:26:50.153319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:22.230155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:22.230155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149814+0000","last_change":"2024-03-24T11:26:47.560502+0000","last_active":"2024-03-24T11:26:50.149814+0000","last_peered":"2024-03-24T11:26:50.149814+0000","last_clean":"2024-03-24T11:26:50.149814+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:26:50.149814+0000","last_undegraded":"2024-03-24T11:26:50.149814+0000","last_fullsized":"2024-03-24T11:26:50.149814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:47.560381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.560381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.212240+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.178657+0000","last_change":"2024-03-24T11:26:28.556498+0000","last_active":"2024-03-24T11:26:50.178657+0000","last_peered":"2024-03-24T11:26:50.178657+0000","last_clean":"2024-03-24T11:26:50.178657+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:26:50.178657+0000","last_undegraded":"2024-03-24T11:26:50.178657+0000","last_fullsized":"2024-03-24T11:26:50.178657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.556368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.556368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:33.125296+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"87'6","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164136+0000","last_change":"2024-03-24T11:22:58.803535+0000","last_active":"2024-03-24T11:26:50.164136+0000","last_peered":"2024-03-24T11:26:50.164136+0000","last_clean":"2024-03-24T11:26:50.164136+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:26:50.164136+0000","last_undegraded":"2024-03-24T11:26:50.164136+0000","last_fullsized":"2024-03-24T11:26:50.164136+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'6","last_scrub_stamp":"2024-03-24T11:22:58.803377+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:22:58.803377+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157968+0000","last_change":"2024-03-24T11:24:46.271791+0000","last_active":"2024-03-24T11:26:50.157968+0000","last_peered":"2024-03-24T11:26:50.157968+0000","last_clean":"2024-03-24T11:26:50.157968+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:26:50.157968+0000","last_undegraded":"2024-03-24T11:26:50.157968+0000","last_fullsized":"2024-03-24T11:26:50.157968+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:24:46.271695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:24:46.271695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.152866+0000","last_change":"2024-03-24T11:25:38.219467+0000","last_active":"2024-03-24T11:26:50.152866+0000","last_peered":"2024-03-24T11:26:50.152866+0000","last_clean":"2024-03-24T11:26:50.152866+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:26:50.152866+0000","last_undegraded":"2024-03-24T11:26:50.152866+0000","last_fullsized":"2024-03-24T11:26:50.152866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:38.219347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.219347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.148706+0000","last_change":"2024-03-24T11:26:30.097875+0000","last_active":"2024-03-24T11:26:50.148706+0000","last_peered":"2024-03-24T11:26:50.148706+0000","last_clean":"2024-03-24T11:26:50.148706+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:26:50.148706+0000","last_undegraded":"2024-03-24T11:26:50.148706+0000","last_fullsized":"2024-03-24T11:26:50.148706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:30.097693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:30.097693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:32.045056+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157984+0000","last_change":"2024-03-24T11:24:29.091627+0000","last_active":"2024-03-24T11:26:50.157984+0000","last_peered":"2024-03-24T11:26:50.157984+0000","last_clean":"2024-03-24T11:26:50.157984+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:26:50.157984+0000","last_undegraded":"2024-03-24T11:26:50.157984+0000","last_fullsized":"2024-03-24T11:26:50.157984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:24:29.091502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:24:29.091502+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.170214+0000","last_change":"2024-03-24T11:25:27.393223+0000","last_active":"2024-03-24T11:26:50.170214+0000","last_peered":"2024-03-24T11:26:50.170214+0000","last_clean":"2024-03-24T11:26:50.170214+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:26:50.170214+0000","last_undegraded":"2024-03-24T11:26:50.170214+0000","last_fullsized":"2024-03-24T11:26:50.170214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:27.393103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:27.393103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157668+0000","last_change":"2024-03-24T11:22:52.829399+0000","last_active":"2024-03-24T11:26:50.157668+0000","last_peered":"2024-03-24T11:26:50.157668+0000","last_clean":"2024-03-24T11:26:50.157668+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:26:50.157668+0000","last_undegraded":"2024-03-24T11:26:50.157668+0000","last_fullsized":"2024-03-24T11:26:50.157668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:22:52.829170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:22:52.829170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.154764+0000","last_change":"2024-03-24T11:25:37.212084+0000","last_active":"2024-03-24T11:26:50.154764+0000","last_peered":"2024-03-24T11:26:50.154764+0000","last_clean":"2024-03-24T11:26:50.154764+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:26:50.154764+0000","last_undegraded":"2024-03-24T11:26:50.154764+0000","last_fullsized":"2024-03-24T11:26:50.154764+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:37.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:37.212008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164945+0000","last_change":"2024-03-24T11:25:36.929044+0000","last_active":"2024-03-24T11:26:50.164945+0000","last_peered":"2024-03-24T11:26:50.164945+0000","last_clean":"2024-03-24T11:26:50.164945+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:26:50.164945+0000","last_undegraded":"2024-03-24T11:26:50.164945+0000","last_fullsized":"2024-03-24T11:26:50.164945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:25:36.928024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.928024+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:51.717916+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157633+0000","last_change":"2024-03-24T11:26:33.238031+0000","last_active":"2024-03-24T11:26:50.157633+0000","last_peered":"2024-03-24T11:26:50.157633+0000","last_clean":"2024-03-24T11:26:50.157633+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:26:50.157633+0000","last_undegraded":"2024-03-24T11:26:50.157633+0000","last_fullsized":"2024-03-24T11:26:50.157633+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.237864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.237864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:40.306806+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164764+0000","last_change":"2024-03-24T11:25:50.982388+0000","last_active":"2024-03-24T11:26:50.164764+0000","last_peered":"2024-03-24T11:26:50.164764+0000","last_clean":"2024-03-24T11:26:50.164764+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:26:50.164764+0000","last_undegraded":"2024-03-24T11:26:50.164764+0000","last_fullsized":"2024-03-24T11:26:50.164764+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:50.982261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:50.982261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.410308+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165540+0000","last_change":"2024-03-24T11:26:33.874445+0000","last_active":"2024-03-24T11:26:50.165540+0000","last_peered":"2024-03-24T11:26:50.165540+0000","last_clean":"2024-03-24T11:26:50.165540+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:26:50.165540+0000","last_undegraded":"2024-03-24T11:26:50.165540+0000","last_fullsized":"2024-03-24T11:26:50.165540+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.874331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.874331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:01.789419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"129'16","reported_seq":292,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165327+0000","last_change":"2024-03-24T11:25:50.138665+0000","last_active":"2024-03-24T11:26:50.165327+0000","last_peered":"2024-03-24T11:26:50.165327+0000","last_clean":"2024-03-24T11:26:50.165327+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:26:50.165327+0000","last_undegraded":"2024-03-24T11:26:50.165327+0000","last_fullsized":"2024-03-24T11:26:50.165327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'16","last_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:25:50.138485+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:04.120931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.181341+0000","last_change":"2024-03-24T11:25:43.229826+0000","last_active":"2024-03-24T11:26:50.181341+0000","last_peered":"2024-03-24T11:26:50.181341+0000","last_clean":"2024-03-24T11:26:50.181341+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:26:50.181341+0000","last_undegraded":"2024-03-24T11:26:50.181341+0000","last_fullsized":"2024-03-24T11:26:50.181341+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:43.229470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.229470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:53.685360+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":369,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149402+0000","last_change":"2024-03-24T11:25:25.010482+0000","last_active":"2024-03-24T11:26:50.149402+0000","last_peered":"2024-03-24T11:26:50.149402+0000","last_clean":"2024-03-24T11:26:50.149402+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:26:50.149402+0000","last_undegraded":"2024-03-24T11:26:50.149402+0000","last_fullsized":"2024-03-24T11:26:50.149402+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:25.010351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:25.010351+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:57.989661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153532+0000","last_change":"2024-03-24T11:25:30.234465+0000","last_active":"2024-03-24T11:26:50.153532+0000","last_peered":"2024-03-24T11:26:50.153532+0000","last_clean":"2024-03-24T11:26:50.153532+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:26:50.153532+0000","last_undegraded":"2024-03-24T11:26:50.153532+0000","last_fullsized":"2024-03-24T11:26:50.153532+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:30.234343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:30.234343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:55.830556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":267,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.167572+0000","last_change":"2024-03-24T11:23:48.940813+0000","last_active":"2024-03-24T11:26:50.167572+0000","last_peered":"2024-03-24T11:26:50.167572+0000","last_clean":"2024-03-24T11:26:50.167572+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:26:50.167572+0000","last_undegraded":"2024-03-24T11:26:50.167572+0000","last_fullsized":"2024-03-24T11:26:50.167572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:23:48.940558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:48.940558+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165671+0000","last_change":"2024-03-24T11:26:39.812393+0000","last_active":"2024-03-24T11:26:50.165671+0000","last_peered":"2024-03-24T11:26:50.165671+0000","last_clean":"2024-03-24T11:26:50.165671+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:26:50.165671+0000","last_undegraded":"2024-03-24T11:26:50.165671+0000","last_fullsized":"2024-03-24T11:26:50.165671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.812259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.728524+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":361,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.166434+0000","last_change":"2024-03-24T11:25:49.322508+0000","last_active":"2024-03-24T11:26:50.166434+0000","last_peered":"2024-03-24T11:26:50.166434+0000","last_clean":"2024-03-24T11:26:50.166434+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:26:50.166434+0000","last_undegraded":"2024-03-24T11:26:50.166434+0000","last_fullsized":"2024-03-24T11:26:50.166434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:49.322383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.322383+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:57.829178+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.176798+0000","last_change":"2024-03-24T11:26:46.640485+0000","last_active":"2024-03-24T11:26:50.176798+0000","last_peered":"2024-03-24T11:26:50.176798+0000","last_clean":"2024-03-24T11:26:50.176798+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:26:50.176798+0000","last_undegraded":"2024-03-24T11:26:50.176798+0000","last_fullsized":"2024-03-24T11:26:50.176798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.640304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.327035+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149551+0000","last_change":"2024-03-24T11:25:39.060762+0000","last_active":"2024-03-24T11:26:50.149551+0000","last_peered":"2024-03-24T11:26:50.149551+0000","last_clean":"2024-03-24T11:26:50.149551+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:26:50.149551+0000","last_undegraded":"2024-03-24T11:26:50.149551+0000","last_fullsized":"2024-03-24T11:26:50.149551+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:25:39.058804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.058804+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.156619+0000","last_change":"2024-03-24T11:26:35.783770+0000","last_active":"2024-03-24T11:26:50.156619+0000","last_peered":"2024-03-24T11:26:50.156619+0000","last_clean":"2024-03-24T11:26:50.156619+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:26:50.156619+0000","last_undegraded":"2024-03-24T11:26:50.156619+0000","last_fullsized":"2024-03-24T11:26:50.156619+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:35.783364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:35.783364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.529071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.160743+0000","last_change":"2024-03-24T11:26:42.302143+0000","last_active":"2024-03-24T11:26:50.160743+0000","last_peered":"2024-03-24T11:26:50.160743+0000","last_clean":"2024-03-24T11:26:50.160743+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:26:50.160743+0000","last_undegraded":"2024-03-24T11:26:50.160743+0000","last_fullsized":"2024-03-24T11:26:50.160743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.301950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:55.496626+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153003+0000","last_change":"2024-03-24T11:26:27.880943+0000","last_active":"2024-03-24T11:26:50.153003+0000","last_peered":"2024-03-24T11:26:50.153003+0000","last_clean":"2024-03-24T11:26:50.153003+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:26:50.153003+0000","last_undegraded":"2024-03-24T11:26:50.153003+0000","last_fullsized":"2024-03-24T11:26:50.153003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:27.880817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.880817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:43.623251+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":288,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.181717+0000","last_change":"2024-03-24T11:25:38.225135+0000","last_active":"2024-03-24T11:26:50.181717+0000","last_peered":"2024-03-24T11:26:50.181717+0000","last_clean":"2024-03-24T11:26:50.181717+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:26:50.181717+0000","last_undegraded":"2024-03-24T11:26:50.181717+0000","last_fullsized":"2024-03-24T11:26:50.181717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:25:38.225011+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.225011+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:06.845575+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153886+0000","last_change":"2024-03-24T11:25:42.250783+0000","last_active":"2024-03-24T11:26:50.153886+0000","last_peered":"2024-03-24T11:26:50.153886+0000","last_clean":"2024-03-24T11:26:50.153886+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:26:50.153886+0000","last_undegraded":"2024-03-24T11:26:50.153886+0000","last_fullsized":"2024-03-24T11:26:50.153886+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:42.250472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:42.250472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149245+0000","last_change":"2024-03-24T11:25:26.007618+0000","last_active":"2024-03-24T11:26:50.149245+0000","last_peered":"2024-03-24T11:26:50.149245+0000","last_clean":"2024-03-24T11:26:50.149245+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:26:50.149245+0000","last_undegraded":"2024-03-24T11:26:50.149245+0000","last_fullsized":"2024-03-24T11:26:50.149245+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:26.007503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:26.007503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:50.412928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149086+0000","last_change":"2024-03-24T11:26:30.706204+0000","last_active":"2024-03-24T11:26:50.149086+0000","last_peered":"2024-03-24T11:26:50.149086+0000","last_clean":"2024-03-24T11:26:50.149086+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:26:50.149086+0000","last_undegraded":"2024-03-24T11:26:50.149086+0000","last_fullsized":"2024-03-24T11:26:50.149086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:30.706071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:30.706071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:50.611182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":261,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.160815+0000","last_change":"2024-03-24T11:23:38.927421+0000","last_active":"2024-03-24T11:26:50.160815+0000","last_peered":"2024-03-24T11:26:50.160815+0000","last_clean":"2024-03-24T11:26:50.160815+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:26:50.160815+0000","last_undegraded":"2024-03-24T11:26:50.160815+0000","last_fullsized":"2024-03-24T11:26:50.160815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-03-24T11:23:38.927200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:38.927200+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"179'8975","reported_seq":9043,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.157689+0000","last_change":"2024-03-24T11:25:02.677444+0000","last_active":"2024-03-24T11:26:58.157689+0000","last_peered":"2024-03-24T11:26:58.157689+0000","last_clean":"2024-03-24T11:26:58.157689+0000","last_became_active":"2024-03-24T11:25:02.676873+0000","last_became_peered":"2024-03-24T11:25:02.676873+0000","last_unstale":"2024-03-24T11:26:58.157689+0000","last_undegraded":"2024-03-24T11:26:58.157689+0000","last_fullsized":"2024-03-24T11:26:58.157689+0000","mapping_epoch":150,"log_start":"166'3900","ondisk_log_start":"166'3900","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462749696,"num_objects":7061,"num_object_clones":0,"num_object_copies":21183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":513152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.152526+0000","last_change":"2024-03-24T11:25:21.100395+0000","last_active":"2024-03-24T11:26:50.152526+0000","last_peered":"2024-03-24T11:26:50.152526+0000","last_clean":"2024-03-24T11:26:50.152526+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:26:50.152526+0000","last_undegraded":"2024-03-24T11:26:50.152526+0000","last_fullsized":"2024-03-24T11:26:50.152526+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:21.100253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:21.100253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":286,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.154168+0000","last_change":"2024-03-24T11:26:34.845593+0000","last_active":"2024-03-24T11:26:50.154168+0000","last_peered":"2024-03-24T11:26:50.154168+0000","last_clean":"2024-03-24T11:26:50.154168+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:26:50.154168+0000","last_undegraded":"2024-03-24T11:26:50.154168+0000","last_fullsized":"2024-03-24T11:26:50.154168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:26:34.845460+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.845460+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.187113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"129'22","reported_seq":300,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.151991+0000","last_change":"2024-03-24T11:25:38.096113+0000","last_active":"2024-03-24T11:26:50.151991+0000","last_peered":"2024-03-24T11:26:50.151991+0000","last_clean":"2024-03-24T11:26:50.151991+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:26:50.151991+0000","last_undegraded":"2024-03-24T11:26:50.151991+0000","last_fullsized":"2024-03-24T11:26:50.151991+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'22","last_scrub_stamp":"2024-03-24T11:25:38.096027+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.096027+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":369,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153266+0000","last_change":"2024-03-24T11:26:48.966305+0000","last_active":"2024-03-24T11:26:50.153266+0000","last_peered":"2024-03-24T11:26:50.153266+0000","last_clean":"2024-03-24T11:26:50.153266+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:26:50.153266+0000","last_undegraded":"2024-03-24T11:26:50.153266+0000","last_fullsized":"2024-03-24T11:26:50.153266+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:48.966187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.966187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.360272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"179'9018","reported_seq":9173,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.197075+0000","last_change":"2024-03-24T11:26:46.326986+0000","last_active":"2024-03-24T11:26:58.197075+0000","last_peered":"2024-03-24T11:26:58.197075+0000","last_clean":"2024-03-24T11:26:58.197075+0000","last_became_active":"2024-03-24T11:25:02.675760+0000","last_became_peered":"2024-03-24T11:25:02.675760+0000","last_unstale":"2024-03-24T11:26:58.197075+0000","last_undegraded":"2024-03-24T11:26:58.197075+0000","last_fullsized":"2024-03-24T11:26:58.197075+0000","mapping_epoch":150,"log_start":"163'3300","ondisk_log_start":"163'3300","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'8169","last_scrub_stamp":"2024-03-24T11:26:46.326655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.326655+0000","objects_scrubbed":7921,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.124923+0000","scrub_duration":1769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460062720,"num_objects":7020,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":244,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157868+0000","last_change":"2024-03-24T11:23:41.875822+0000","last_active":"2024-03-24T11:26:50.157868+0000","last_peered":"2024-03-24T11:26:50.157868+0000","last_clean":"2024-03-24T11:26:50.157868+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:26:50.157868+0000","last_undegraded":"2024-03-24T11:26:50.157868+0000","last_fullsized":"2024-03-24T11:26:50.157868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:41.875525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:41.875525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.154387+0000","last_change":"2024-03-24T11:25:36.259283+0000","last_active":"2024-03-24T11:26:50.154387+0000","last_peered":"2024-03-24T11:26:50.154387+0000","last_clean":"2024-03-24T11:26:50.154387+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:26:50.154387+0000","last_undegraded":"2024-03-24T11:26:50.154387+0000","last_fullsized":"2024-03-24T11:26:50.154387+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:36.259040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.259040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:57.938648+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":291,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165298+0000","last_change":"2024-03-24T11:26:36.876595+0000","last_active":"2024-03-24T11:26:50.165298+0000","last_peered":"2024-03-24T11:26:50.165298+0000","last_clean":"2024-03-24T11:26:50.165298+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:26:50.165298+0000","last_undegraded":"2024-03-24T11:26:50.165298+0000","last_fullsized":"2024-03-24T11:26:50.165298+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:26:36.876465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.876465+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:37.974940+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.156800+0000","last_change":"2024-03-24T11:26:37.862505+0000","last_active":"2024-03-24T11:26:50.156800+0000","last_peered":"2024-03-24T11:26:50.156800+0000","last_clean":"2024-03-24T11:26:50.156800+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:26:50.156800+0000","last_undegraded":"2024-03-24T11:26:50.156800+0000","last_fullsized":"2024-03-24T11:26:50.156800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.862389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.862389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.892368+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"179'9141","reported_seq":9209,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.896503+0000","last_change":"2024-03-24T11:25:02.676467+0000","last_active":"2024-03-24T11:26:58.896503+0000","last_peered":"2024-03-24T11:26:58.896503+0000","last_clean":"2024-03-24T11:26:58.896503+0000","last_became_active":"2024-03-24T11:25:02.676144+0000","last_became_peered":"2024-03-24T11:25:02.676144+0000","last_unstale":"2024-03-24T11:26:58.896503+0000","last_undegraded":"2024-03-24T11:26:58.896503+0000","last_fullsized":"2024-03-24T11:26:58.896503+0000","mapping_epoch":150,"log_start":"164'3600","ondisk_log_start":"164'3600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:02.442897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453836800,"num_objects":6925,"num_object_clones":0,"num_object_copies":20775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":514112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.152614+0000","last_change":"2024-03-24T11:25:33.186360+0000","last_active":"2024-03-24T11:26:50.152614+0000","last_peered":"2024-03-24T11:26:50.152614+0000","last_clean":"2024-03-24T11:26:50.152614+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:26:50.152614+0000","last_undegraded":"2024-03-24T11:26:50.152614+0000","last_fullsized":"2024-03-24T11:26:50.152614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:33.186253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:33.186253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:01.614006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.155849+0000","last_change":"2024-03-24T11:26:41.913283+0000","last_active":"2024-03-24T11:26:50.155849+0000","last_peered":"2024-03-24T11:26:50.155849+0000","last_clean":"2024-03-24T11:26:50.155849+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:26:50.155849+0000","last_undegraded":"2024-03-24T11:26:50.155849+0000","last_fullsized":"2024-03-24T11:26:50.155849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.913165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.913165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.067615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.154829+0000","last_change":"2024-03-24T11:26:44.921628+0000","last_active":"2024-03-24T11:26:50.154829+0000","last_peered":"2024-03-24T11:26:50.154829+0000","last_clean":"2024-03-24T11:26:50.154829+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:26:50.154829+0000","last_undegraded":"2024-03-24T11:26:50.154829+0000","last_fullsized":"2024-03-24T11:26:50.154829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:44.921512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:44.921512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:52.518225+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"88'28","reported_seq":316,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.150001+0000","last_change":"2024-03-24T11:26:34.694013+0000","last_active":"2024-03-24T11:26:50.150001+0000","last_peered":"2024-03-24T11:26:50.150001+0000","last_clean":"2024-03-24T11:26:50.150001+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:26:50.150001+0000","last_undegraded":"2024-03-24T11:26:50.150001+0000","last_fullsized":"2024-03-24T11:26:50.150001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'28","last_scrub_stamp":"2024-03-24T11:26:34.693862+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.693862+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:54.976521+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"179'8939","reported_seq":9095,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.410889+0000","last_change":"2024-03-24T11:26:44.358566+0000","last_active":"2024-03-24T11:26:58.410889+0000","last_peered":"2024-03-24T11:26:58.410889+0000","last_clean":"2024-03-24T11:26:58.410889+0000","last_became_active":"2024-03-24T11:25:02.599484+0000","last_became_peered":"2024-03-24T11:25:02.599484+0000","last_unstale":"2024-03-24T11:26:58.410889+0000","last_undegraded":"2024-03-24T11:26:58.410889+0000","last_fullsized":"2024-03-24T11:26:58.410889+0000","mapping_epoch":150,"log_start":"166'4100","ondisk_log_start":"166'4100","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7990","last_scrub_stamp":"2024-03-24T11:26:44.358125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:44.358125+0000","objects_scrubbed":7882,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:08.519779+0000","scrub_duration":3569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459603968,"num_objects":7013,"num_object_clones":0,"num_object_copies":21039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8939,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.173055+0000","last_change":"2024-03-24T11:26:26.644711+0000","last_active":"2024-03-24T11:26:50.173055+0000","last_peered":"2024-03-24T11:26:50.173055+0000","last_clean":"2024-03-24T11:26:50.173055+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:26:50.173055+0000","last_undegraded":"2024-03-24T11:26:50.173055+0000","last_fullsized":"2024-03-24T11:26:50.173055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.644597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.644597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.058406+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157163+0000","last_change":"2024-03-24T11:26:26.849730+0000","last_active":"2024-03-24T11:26:50.157163+0000","last_peered":"2024-03-24T11:26:50.157163+0000","last_clean":"2024-03-24T11:26:50.157163+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:26:50.157163+0000","last_undegraded":"2024-03-24T11:26:50.157163+0000","last_fullsized":"2024-03-24T11:26:50.157163+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.849557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.849557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:50.544052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":278,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.148982+0000","last_change":"2024-03-24T11:25:33.078676+0000","last_active":"2024-03-24T11:26:50.148982+0000","last_peered":"2024-03-24T11:26:50.148982+0000","last_clean":"2024-03-24T11:26:50.148982+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:26:50.148982+0000","last_undegraded":"2024-03-24T11:26:50.148982+0000","last_fullsized":"2024-03-24T11:26:50.148982+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:25:33.078550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:33.078550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:53.958853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.177169+0000","last_change":"2024-03-24T11:26:29.507764+0000","last_active":"2024-03-24T11:26:50.177169+0000","last_peered":"2024-03-24T11:26:50.177169+0000","last_clean":"2024-03-24T11:26:50.177169+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:26:50.177169+0000","last_undegraded":"2024-03-24T11:26:50.177169+0000","last_fullsized":"2024-03-24T11:26:50.177169+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:26:29.507604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.507604+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:38.038415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"179'9204","reported_seq":9277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.149190+0000","last_change":"2024-03-24T11:26:50.167830+0000","last_active":"2024-03-24T11:26:58.149190+0000","last_peered":"2024-03-24T11:26:58.149190+0000","last_clean":"2024-03-24T11:26:58.149190+0000","last_became_active":"2024-03-24T11:25:02.675496+0000","last_became_peered":"2024-03-24T11:25:02.675496+0000","last_unstale":"2024-03-24T11:26:58.149190+0000","last_undegraded":"2024-03-24T11:26:58.149190+0000","last_fullsized":"2024-03-24T11:26:58.149190+0000","mapping_epoch":150,"log_start":"166'4200","ondisk_log_start":"166'4200","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":199,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461766656,"num_objects":7046,"num_object_clones":0,"num_object_copies":21138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9204,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":402,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.148600+0000","last_change":"2024-03-24T11:25:48.067965+0000","last_active":"2024-03-24T11:26:50.148600+0000","last_peered":"2024-03-24T11:26:50.148600+0000","last_clean":"2024-03-24T11:26:50.148600+0000","last_became_active":"2024-03-24T11:24:29.093656+0000","last_became_peered":"2024-03-24T11:24:29.093656+0000","last_unstale":"2024-03-24T11:26:50.148600+0000","last_undegraded":"2024-03-24T11:26:50.148600+0000","last_fullsized":"2024-03-24T11:26:50.148600+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:48.067828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:48.067828+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.158043+0000","last_change":"2024-03-24T11:26:40.231258+0000","last_active":"2024-03-24T11:26:50.158043+0000","last_peered":"2024-03-24T11:26:50.158043+0000","last_clean":"2024-03-24T11:26:50.158043+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:26:50.158043+0000","last_undegraded":"2024-03-24T11:26:50.158043+0000","last_fullsized":"2024-03-24T11:26:50.158043+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:40.231135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:40.231135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:43.325157+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":283,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.152278+0000","last_change":"2024-03-24T11:25:35.099894+0000","last_active":"2024-03-24T11:26:50.152278+0000","last_peered":"2024-03-24T11:26:50.152278+0000","last_clean":"2024-03-24T11:26:50.152278+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:26:50.152278+0000","last_undegraded":"2024-03-24T11:26:50.152278+0000","last_fullsized":"2024-03-24T11:26:50.152278+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:25:35.099746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.099746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:58.528633+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165496+0000","last_change":"2024-03-24T11:26:31.873008+0000","last_active":"2024-03-24T11:26:50.165496+0000","last_peered":"2024-03-24T11:26:50.165496+0000","last_clean":"2024-03-24T11:26:50.165496+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:26:50.165496+0000","last_undegraded":"2024-03-24T11:26:50.165496+0000","last_fullsized":"2024-03-24T11:26:50.165496+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:31.872524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:31.872524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:40.048777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":413,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165597+0000","last_change":"2024-03-24T11:25:26.016622+0000","last_active":"2024-03-24T11:26:50.165597+0000","last_peered":"2024-03-24T11:26:50.165597+0000","last_clean":"2024-03-24T11:26:50.165597+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:26:50.165597+0000","last_undegraded":"2024-03-24T11:26:50.165597+0000","last_fullsized":"2024-03-24T11:26:50.165597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:25:26.016381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:26.016381+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:56.776323+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.178226+0000","last_change":"2024-03-24T11:26:38.524661+0000","last_active":"2024-03-24T11:26:50.178226+0000","last_peered":"2024-03-24T11:26:50.178226+0000","last_clean":"2024-03-24T11:26:50.178226+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:26:50.178226+0000","last_undegraded":"2024-03-24T11:26:50.178226+0000","last_fullsized":"2024-03-24T11:26:50.178226+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:38.524541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:38.524541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.968194+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"179'9242","reported_seq":9310,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.614066+0000","last_change":"2024-03-24T11:25:02.601118+0000","last_active":"2024-03-24T11:26:58.614066+0000","last_peered":"2024-03-24T11:26:58.614066+0000","last_clean":"2024-03-24T11:26:58.614066+0000","last_became_active":"2024-03-24T11:25:02.600777+0000","last_became_peered":"2024-03-24T11:25:02.600777+0000","last_unstale":"2024-03-24T11:26:58.614066+0000","last_undegraded":"2024-03-24T11:26:58.614066+0000","last_fullsized":"2024-03-24T11:26:58.614066+0000","mapping_epoch":150,"log_start":"169'4700","ondisk_log_start":"169'4700","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:03.387043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472645632,"num_objects":7212,"num_object_clones":0,"num_object_copies":21636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9242,"num_write_kb":526528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":542,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153326+0000","last_change":"2024-03-24T11:25:49.282133+0000","last_active":"2024-03-24T11:26:50.153326+0000","last_peered":"2024-03-24T11:26:50.153326+0000","last_clean":"2024-03-24T11:26:50.153326+0000","last_became_active":"2024-03-24T11:24:29.092038+0000","last_became_peered":"2024-03-24T11:24:29.092038+0000","last_unstale":"2024-03-24T11:26:50.153326+0000","last_undegraded":"2024-03-24T11:26:50.153326+0000","last_fullsized":"2024-03-24T11:26:50.153326+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:49.282048+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:25:49.282048+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.282048+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:02.192420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.154892+0000","last_change":"2024-03-24T11:25:44.288277+0000","last_active":"2024-03-24T11:26:50.154892+0000","last_peered":"2024-03-24T11:26:50.154892+0000","last_clean":"2024-03-24T11:26:50.154892+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:26:50.154892+0000","last_undegraded":"2024-03-24T11:26:50.154892+0000","last_fullsized":"2024-03-24T11:26:50.154892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:25:44.288152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:57.902922+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":281,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.174383+0000","last_change":"2024-03-24T11:26:25.702207+0000","last_active":"2024-03-24T11:26:50.174383+0000","last_peered":"2024-03-24T11:26:50.174383+0000","last_clean":"2024-03-24T11:26:50.174383+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:26:50.174383+0000","last_undegraded":"2024-03-24T11:26:50.174383+0000","last_fullsized":"2024-03-24T11:26:50.174383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:25.702101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:25.702101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:32.898822+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":281,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.156101+0000","last_change":"2024-03-24T11:25:36.710275+0000","last_active":"2024-03-24T11:26:50.156101+0000","last_peered":"2024-03-24T11:26:50.156101+0000","last_clean":"2024-03-24T11:26:50.156101+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:26:50.156101+0000","last_undegraded":"2024-03-24T11:26:50.156101+0000","last_fullsized":"2024-03-24T11:26:50.156101+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:36.710123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.710123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:03.110264+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":270,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.160355+0000","last_change":"2024-03-24T11:25:45.765628+0000","last_active":"2024-03-24T11:26:50.160355+0000","last_peered":"2024-03-24T11:26:50.160355+0000","last_clean":"2024-03-24T11:26:50.160355+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:26:50.160355+0000","last_undegraded":"2024-03-24T11:26:50.160355+0000","last_fullsized":"2024-03-24T11:26:50.160355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:25:45.764917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:45.764917+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:08.852649+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.156082+0000","last_change":"2024-03-24T11:26:29.820470+0000","last_active":"2024-03-24T11:26:50.156082+0000","last_peered":"2024-03-24T11:26:50.156082+0000","last_clean":"2024-03-24T11:26:50.156082+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:26:50.156082+0000","last_undegraded":"2024-03-24T11:26:50.156082+0000","last_fullsized":"2024-03-24T11:26:50.156082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:29.820240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.820240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:41.262557+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"179'8832","reported_seq":8987,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.182798+0000","last_change":"2024-03-24T11:26:48.313995+0000","last_active":"2024-03-24T11:26:58.182798+0000","last_peered":"2024-03-24T11:26:58.182798+0000","last_clean":"2024-03-24T11:26:58.182798+0000","last_became_active":"2024-03-24T11:25:02.600137+0000","last_became_peered":"2024-03-24T11:25:02.600137+0000","last_unstale":"2024-03-24T11:26:58.182798+0000","last_undegraded":"2024-03-24T11:26:58.182798+0000","last_fullsized":"2024-03-24T11:26:58.182798+0000","mapping_epoch":150,"log_start":"161'3100","ondisk_log_start":"161'3100","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8126","last_scrub_stamp":"2024-03-24T11:26:48.313801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.313801+0000","objects_scrubbed":7677,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.465833+0000","scrub_duration":2728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451280896,"num_objects":6886,"num_object_clones":0,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.148850+0000","last_change":"2024-03-24T11:26:49.074595+0000","last_active":"2024-03-24T11:26:50.148850+0000","last_peered":"2024-03-24T11:26:50.148850+0000","last_clean":"2024-03-24T11:26:50.148850+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:26:50.148850+0000","last_undegraded":"2024-03-24T11:26:50.148850+0000","last_fullsized":"2024-03-24T11:26:50.148850+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.074485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:10.184136+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":535,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.152741+0000","last_change":"2024-03-24T11:26:36.872094+0000","last_active":"2024-03-24T11:26:50.152741+0000","last_peered":"2024-03-24T11:26:50.152741+0000","last_clean":"2024-03-24T11:26:50.152741+0000","last_became_active":"2024-03-24T11:24:23.650976+0000","last_became_peered":"2024-03-24T11:24:23.650976+0000","last_unstale":"2024-03-24T11:26:50.152741+0000","last_undegraded":"2024-03-24T11:26:50.152741+0000","last_fullsized":"2024-03-24T11:26:50.152741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:36.871960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.871960+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:41.364084+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.159994+0000","last_change":"2024-03-24T11:26:35.227647+0000","last_active":"2024-03-24T11:26:50.159994+0000","last_peered":"2024-03-24T11:26:50.159994+0000","last_clean":"2024-03-24T11:26:50.159994+0000","last_became_active":"2024-03-24T11:18:53.678099+0000","last_became_peered":"2024-03-24T11:18:53.678099+0000","last_unstale":"2024-03-24T11:26:50.159994+0000","last_undegraded":"2024-03-24T11:26:50.159994+0000","last_fullsized":"2024-03-24T11:26:50.159994+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:35.227538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:35.227538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:03.040853+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"176'24","reported_seq":403,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.150392+0000","last_change":"2024-03-24T11:26:37.676875+0000","last_active":"2024-03-24T11:26:50.150392+0000","last_peered":"2024-03-24T11:26:50.150392+0000","last_clean":"2024-03-24T11:26:50.150392+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:26:50.150392+0000","last_undegraded":"2024-03-24T11:26:50.150392+0000","last_fullsized":"2024-03-24T11:26:50.150392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'24","last_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.676764+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:48.350269+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.154268+0000","last_change":"2024-03-24T11:25:40.290783+0000","last_active":"2024-03-24T11:26:50.154268+0000","last_peered":"2024-03-24T11:26:50.154268+0000","last_clean":"2024-03-24T11:26:50.154268+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:26:50.154268+0000","last_undegraded":"2024-03-24T11:26:50.154268+0000","last_fullsized":"2024-03-24T11:26:50.154268+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.290398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:05.517014+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"179'8944","reported_seq":9098,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.397240+0000","last_change":"2024-03-24T11:26:42.718381+0000","last_active":"2024-03-24T11:26:58.397240+0000","last_peered":"2024-03-24T11:26:58.397240+0000","last_clean":"2024-03-24T11:26:58.397240+0000","last_became_active":"2024-03-24T11:25:02.674959+0000","last_became_peered":"2024-03-24T11:25:02.674959+0000","last_unstale":"2024-03-24T11:26:58.397240+0000","last_undegraded":"2024-03-24T11:26:58.397240+0000","last_fullsized":"2024-03-24T11:26:58.397240+0000","mapping_epoch":150,"log_start":"167'4100","ondisk_log_start":"167'4100","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7858","last_scrub_stamp":"2024-03-24T11:26:42.718210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.718210+0000","objects_scrubbed":7774,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.515083+0000","scrub_duration":2944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453509120,"num_objects":6920,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8944,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":402,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.166191+0000","last_change":"2024-03-24T11:26:27.707698+0000","last_active":"2024-03-24T11:26:50.166191+0000","last_peered":"2024-03-24T11:26:50.166191+0000","last_clean":"2024-03-24T11:26:50.166191+0000","last_became_active":"2024-03-24T11:24:29.087723+0000","last_became_peered":"2024-03-24T11:24:29.087723+0000","last_unstale":"2024-03-24T11:26:50.166191+0000","last_undegraded":"2024-03-24T11:26:50.166191+0000","last_fullsized":"2024-03-24T11:26:50.166191+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:27.707575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.707575+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:30.342451+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164179+0000","last_change":"2024-03-24T11:26:42.779061+0000","last_active":"2024-03-24T11:26:50.164179+0000","last_peered":"2024-03-24T11:26:50.164179+0000","last_clean":"2024-03-24T11:26:50.164179+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:26:50.164179+0000","last_undegraded":"2024-03-24T11:26:50.164179+0000","last_fullsized":"2024-03-24T11:26:50.164179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.778835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.166597+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":275,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.172804+0000","last_change":"2024-03-24T11:26:41.522630+0000","last_active":"2024-03-24T11:26:50.172804+0000","last_peered":"2024-03-24T11:26:50.172804+0000","last_clean":"2024-03-24T11:26:50.172804+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:26:50.172804+0000","last_undegraded":"2024-03-24T11:26:50.172804+0000","last_fullsized":"2024-03-24T11:26:50.172804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.522032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:42.372094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":754,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:57.123824+0000","last_change":"2024-03-24T11:25:27.279271+0000","last_active":"2024-03-24T11:26:57.123824+0000","last_peered":"2024-03-24T11:26:57.123824+0000","last_clean":"2024-03-24T11:26:57.123824+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:26:57.123824+0000","last_undegraded":"2024-03-24T11:26:57.123824+0000","last_fullsized":"2024-03-24T11:26:57.123824+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:25:27.279145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:25:27.279145+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:01.958581+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":477,"num_read_kb":472,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":278,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.181435+0000","last_change":"2024-03-24T11:24:44.362662+0000","last_active":"2024-03-24T11:26:50.181435+0000","last_peered":"2024-03-24T11:26:50.181435+0000","last_clean":"2024-03-24T11:26:50.181435+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:26:50.181435+0000","last_undegraded":"2024-03-24T11:26:50.181435+0000","last_fullsized":"2024-03-24T11:26:50.181435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:24:44.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:24:44.362477+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149209+0000","last_change":"2024-03-24T11:24:44.081237+0000","last_active":"2024-03-24T11:26:50.149209+0000","last_peered":"2024-03-24T11:26:50.149209+0000","last_clean":"2024-03-24T11:26:50.149209+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:26:50.149209+0000","last_undegraded":"2024-03-24T11:26:50.149209+0000","last_fullsized":"2024-03-24T11:26:50.149209+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:24:44.081079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:24:44.081079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"179'9032","reported_seq":9100,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.612663+0000","last_change":"2024-03-24T11:25:02.677146+0000","last_active":"2024-03-24T11:26:58.612663+0000","last_peered":"2024-03-24T11:26:58.612663+0000","last_clean":"2024-03-24T11:26:58.612663+0000","last_became_active":"2024-03-24T11:25:02.676840+0000","last_became_peered":"2024-03-24T11:25:02.676840+0000","last_unstale":"2024-03-24T11:26:58.612663+0000","last_undegraded":"2024-03-24T11:26:58.612663+0000","last_fullsized":"2024-03-24T11:26:58.612663+0000","mapping_epoch":150,"log_start":"168'4500","ondisk_log_start":"168'4500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:03.387128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463994880,"num_objects":7080,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9032,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":408,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.150132+0000","last_change":"2024-03-24T11:26:43.595952+0000","last_active":"2024-03-24T11:26:50.150132+0000","last_peered":"2024-03-24T11:26:50.150132+0000","last_clean":"2024-03-24T11:26:50.150132+0000","last_became_active":"2024-03-24T11:24:29.081667+0000","last_became_peered":"2024-03-24T11:24:29.081667+0000","last_unstale":"2024-03-24T11:26:50.150132+0000","last_undegraded":"2024-03-24T11:26:50.150132+0000","last_fullsized":"2024-03-24T11:26:50.150132+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:43.595832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.595832+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.040167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.176398+0000","last_change":"2024-03-24T11:25:49.059451+0000","last_active":"2024-03-24T11:26:50.176398+0000","last_peered":"2024-03-24T11:26:50.176398+0000","last_clean":"2024-03-24T11:26:50.176398+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:26:50.176398+0000","last_undegraded":"2024-03-24T11:26:50.176398+0000","last_fullsized":"2024-03-24T11:26:50.176398+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:49.058968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.058968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:53.423748+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165466+0000","last_change":"2024-03-24T11:25:50.022235+0000","last_active":"2024-03-24T11:26:50.165466+0000","last_peered":"2024-03-24T11:26:50.165466+0000","last_clean":"2024-03-24T11:26:50.165466+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:26:50.165466+0000","last_undegraded":"2024-03-24T11:26:50.165466+0000","last_fullsized":"2024-03-24T11:26:50.165466+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:50.022117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:50.022117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:07.100402+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":296,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.176948+0000","last_change":"2024-03-24T11:25:34.223822+0000","last_active":"2024-03-24T11:26:50.176948+0000","last_peered":"2024-03-24T11:26:50.176948+0000","last_clean":"2024-03-24T11:26:50.176948+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:26:50.176948+0000","last_undegraded":"2024-03-24T11:26:50.176948+0000","last_fullsized":"2024-03-24T11:26:50.176948+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-24T11:25:34.223668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:34.223668+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.150001+0000","last_change":"2024-03-24T11:26:33.660212+0000","last_active":"2024-03-24T11:26:50.150001+0000","last_peered":"2024-03-24T11:26:50.150001+0000","last_clean":"2024-03-24T11:26:50.150001+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:26:50.150001+0000","last_undegraded":"2024-03-24T11:26:50.150001+0000","last_fullsized":"2024-03-24T11:26:50.150001+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.660136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.660136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:34.463306+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"179'9084","reported_seq":9152,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.897016+0000","last_change":"2024-03-24T11:25:02.677293+0000","last_active":"2024-03-24T11:26:58.897016+0000","last_peered":"2024-03-24T11:26:58.897016+0000","last_clean":"2024-03-24T11:26:58.897016+0000","last_became_active":"2024-03-24T11:25:02.676938+0000","last_became_peered":"2024-03-24T11:25:02.676938+0000","last_unstale":"2024-03-24T11:26:58.897016+0000","last_undegraded":"2024-03-24T11:26:58.897016+0000","last_fullsized":"2024-03-24T11:26:58.897016+0000","mapping_epoch":150,"log_start":"164'3600","ondisk_log_start":"164'3600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:02.444674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453771264,"num_objects":6924,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9084,"num_write_kb":512256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":408,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.150466+0000","last_change":"2024-03-24T11:26:32.673765+0000","last_active":"2024-03-24T11:26:50.150466+0000","last_peered":"2024-03-24T11:26:50.150466+0000","last_clean":"2024-03-24T11:26:50.150466+0000","last_became_active":"2024-03-24T11:24:29.117106+0000","last_became_peered":"2024-03-24T11:24:29.117106+0000","last_unstale":"2024-03-24T11:26:50.150466+0000","last_undegraded":"2024-03-24T11:26:50.150466+0000","last_fullsized":"2024-03-24T11:26:50.150466+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:32.673525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.673525+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.552032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":253,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.169687+0000","last_change":"2024-03-24T11:25:28.354779+0000","last_active":"2024-03-24T11:26:50.169687+0000","last_peered":"2024-03-24T11:26:50.169687+0000","last_clean":"2024-03-24T11:26:50.169687+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:26:50.169687+0000","last_undegraded":"2024-03-24T11:26:50.169687+0000","last_fullsized":"2024-03-24T11:26:50.169687+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:25:28.354644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:28.354644+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:53.521879+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":275,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149400+0000","last_change":"2024-03-24T11:25:19.123540+0000","last_active":"2024-03-24T11:26:50.149400+0000","last_peered":"2024-03-24T11:26:50.149400+0000","last_clean":"2024-03-24T11:26:50.149400+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:26:50.149400+0000","last_undegraded":"2024-03-24T11:26:50.149400+0000","last_fullsized":"2024-03-24T11:26:50.149400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:19.123468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:19.123468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":291,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164849+0000","last_change":"2024-03-24T11:26:49.790900+0000","last_active":"2024-03-24T11:26:50.164849+0000","last_peered":"2024-03-24T11:26:50.164849+0000","last_clean":"2024-03-24T11:26:50.164849+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:26:50.164849+0000","last_undegraded":"2024-03-24T11:26:50.164849+0000","last_fullsized":"2024-03-24T11:26:50.164849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:26:49.790753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.790753+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.066698+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":370,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:53.820298+0000","last_change":"2024-03-24T11:25:40.204011+0000","last_active":"2024-03-24T11:26:53.820298+0000","last_peered":"2024-03-24T11:26:53.820298+0000","last_clean":"2024-03-24T11:26:53.820298+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:26:53.820298+0000","last_undegraded":"2024-03-24T11:26:53.820298+0000","last_fullsized":"2024-03-24T11:26:53.820298+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:40.203874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.203874+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:06.158489+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"179'9003","reported_seq":9158,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.138062+0000","last_change":"2024-03-24T11:26:49.441112+0000","last_active":"2024-03-24T11:26:58.138062+0000","last_peered":"2024-03-24T11:26:58.138062+0000","last_clean":"2024-03-24T11:26:58.138062+0000","last_became_active":"2024-03-24T11:25:02.675745+0000","last_became_peered":"2024-03-24T11:25:02.675745+0000","last_unstale":"2024-03-24T11:26:58.138062+0000","last_undegraded":"2024-03-24T11:26:58.138062+0000","last_fullsized":"2024-03-24T11:26:58.138062+0000","mapping_epoch":150,"log_start":"166'4000","ondisk_log_start":"166'4000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8346","last_scrub_stamp":"2024-03-24T11:26:49.440843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.440843+0000","objects_scrubbed":7686,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.854214+0000","scrub_duration":2353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454819864,"num_objects":6941,"num_object_clones":0,"num_object_copies":20823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6941,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9003,"num_write_kb":510145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":394,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.162371+0000","last_change":"2024-03-24T11:24:30.117954+0000","last_active":"2024-03-24T11:26:50.162371+0000","last_peered":"2024-03-24T11:26:50.162371+0000","last_clean":"2024-03-24T11:26:50.162371+0000","last_became_active":"2024-03-24T11:24:29.087468+0000","last_became_peered":"2024-03-24T11:24:29.087468+0000","last_unstale":"2024-03-24T11:26:50.162371+0000","last_undegraded":"2024-03-24T11:26:50.162371+0000","last_fullsized":"2024-03-24T11:26:50.162371+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:24:30.117825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:24:30.117825+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.153666+0000","last_change":"2024-03-24T11:25:26.285547+0000","last_active":"2024-03-24T11:26:50.153666+0000","last_peered":"2024-03-24T11:26:50.153666+0000","last_clean":"2024-03-24T11:26:50.153666+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:26:50.153666+0000","last_undegraded":"2024-03-24T11:26:50.153666+0000","last_fullsized":"2024-03-24T11:26:50.153666+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:26.285465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:26.285465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.181527+0000","last_change":"2024-03-24T11:25:44.174500+0000","last_active":"2024-03-24T11:26:50.181527+0000","last_peered":"2024-03-24T11:26:50.181527+0000","last_clean":"2024-03-24T11:26:50.181527+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:26:50.181527+0000","last_undegraded":"2024-03-24T11:26:50.181527+0000","last_fullsized":"2024-03-24T11:26:50.181527+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:25:44.174271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:44.174271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:07.724718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":281,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165088+0000","last_change":"2024-03-24T11:25:27.130734+0000","last_active":"2024-03-24T11:26:50.165088+0000","last_peered":"2024-03-24T11:26:50.165088+0000","last_clean":"2024-03-24T11:26:50.165088+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:26:50.165088+0000","last_undegraded":"2024-03-24T11:26:50.165088+0000","last_fullsized":"2024-03-24T11:26:50.165088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-03-24T11:25:27.130599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:27.130599+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":353,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164437+0000","last_change":"2024-03-24T11:25:20.232658+0000","last_active":"2024-03-24T11:26:50.164437+0000","last_peered":"2024-03-24T11:26:50.164437+0000","last_clean":"2024-03-24T11:26:50.164437+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:26:50.164437+0000","last_undegraded":"2024-03-24T11:26:50.164437+0000","last_fullsized":"2024-03-24T11:26:50.164437+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:20.232540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:20.232540+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:57.346286+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"179'8858","reported_seq":8925,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.936502+0000","last_change":"2024-03-24T11:25:02.676061+0000","last_active":"2024-03-24T11:26:58.936502+0000","last_peered":"2024-03-24T11:26:58.936502+0000","last_clean":"2024-03-24T11:26:58.936502+0000","last_became_active":"2024-03-24T11:25:02.675706+0000","last_became_peered":"2024-03-24T11:25:02.675706+0000","last_unstale":"2024-03-24T11:26:58.936502+0000","last_undegraded":"2024-03-24T11:26:58.936502+0000","last_fullsized":"2024-03-24T11:26:58.936502+0000","mapping_epoch":150,"log_start":"164'3400","ondisk_log_start":"164'3400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":5458,"log_dups_size":0,"ondisk_log_size":5458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:02.444640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453509120,"num_objects":6920,"num_object_clones":0,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":402,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165424+0000","last_change":"2024-03-24T11:26:29.924513+0000","last_active":"2024-03-24T11:26:50.165424+0000","last_peered":"2024-03-24T11:26:50.165424+0000","last_clean":"2024-03-24T11:26:50.165424+0000","last_became_active":"2024-03-24T11:24:29.082786+0000","last_became_peered":"2024-03-24T11:24:29.082786+0000","last_unstale":"2024-03-24T11:26:50.165424+0000","last_undegraded":"2024-03-24T11:26:50.165424+0000","last_fullsized":"2024-03-24T11:26:50.165424+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:29.924076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.924076+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:39.946530+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.166010+0000","last_change":"2024-03-24T11:25:42.060954+0000","last_active":"2024-03-24T11:26:50.166010+0000","last_peered":"2024-03-24T11:26:50.166010+0000","last_clean":"2024-03-24T11:26:50.166010+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:26:50.166010+0000","last_undegraded":"2024-03-24T11:26:50.166010+0000","last_fullsized":"2024-03-24T11:26:50.166010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:42.060840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:49.118465+0000","last_clean_scrub_stamp":"2024-03-24T11:25:42.060840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.160577+0000","last_change":"2024-03-24T11:26:39.266713+0000","last_active":"2024-03-24T11:26:50.160577+0000","last_peered":"2024-03-24T11:26:50.160577+0000","last_clean":"2024-03-24T11:26:50.160577+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:26:50.160577+0000","last_undegraded":"2024-03-24T11:26:50.160577+0000","last_fullsized":"2024-03-24T11:26:50.160577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.265692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.265692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.928492+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":285,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164342+0000","last_change":"2024-03-24T11:25:35.339628+0000","last_active":"2024-03-24T11:26:50.164342+0000","last_peered":"2024-03-24T11:26:50.164342+0000","last_clean":"2024-03-24T11:26:50.164342+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:26:50.164342+0000","last_undegraded":"2024-03-24T11:26:50.164342+0000","last_fullsized":"2024-03-24T11:26:50.164342+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-24T11:25:35.339383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.339383+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:03.387157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164885+0000","last_change":"2024-03-24T11:26:28.950793+0000","last_active":"2024-03-24T11:26:50.164885+0000","last_peered":"2024-03-24T11:26:50.164885+0000","last_clean":"2024-03-24T11:26:50.164885+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:26:50.164885+0000","last_undegraded":"2024-03-24T11:26:50.164885+0000","last_fullsized":"2024-03-24T11:26:50.164885+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.950670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.950670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:30.361274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"179'9067","reported_seq":9218,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.356507+0000","last_change":"2024-03-24T11:26:34.774580+0000","last_active":"2024-03-24T11:26:58.356507+0000","last_peered":"2024-03-24T11:26:58.356507+0000","last_clean":"2024-03-24T11:26:58.356507+0000","last_became_active":"2024-03-24T11:25:02.600315+0000","last_became_peered":"2024-03-24T11:25:02.600315+0000","last_unstale":"2024-03-24T11:26:58.356507+0000","last_undegraded":"2024-03-24T11:26:58.356507+0000","last_fullsized":"2024-03-24T11:26:58.356507+0000","mapping_epoch":150,"log_start":"167'4200","ondisk_log_start":"167'4200","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7439","last_scrub_stamp":"2024-03-24T11:26:34.774083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.774083+0000","objects_scrubbed":7347,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:52.198850+0000","scrub_duration":2963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459603968,"num_objects":7013,"num_object_clones":0,"num_object_copies":21039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":402,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.152114+0000","last_change":"2024-03-24T11:25:40.019964+0000","last_active":"2024-03-24T11:26:50.152114+0000","last_peered":"2024-03-24T11:26:50.152114+0000","last_clean":"2024-03-24T11:26:50.152114+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:26:50.152114+0000","last_undegraded":"2024-03-24T11:26:50.152114+0000","last_fullsized":"2024-03-24T11:26:50.152114+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:40.019881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.019881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:56.068608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149187+0000","last_change":"2024-03-24T11:25:20.173363+0000","last_active":"2024-03-24T11:26:50.149187+0000","last_peered":"2024-03-24T11:26:50.149187+0000","last_clean":"2024-03-24T11:26:50.149187+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:26:50.149187+0000","last_undegraded":"2024-03-24T11:26:50.149187+0000","last_fullsized":"2024-03-24T11:26:50.149187+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:20.173220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:20.173220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":369,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165830+0000","last_change":"2024-03-24T11:26:42.827014+0000","last_active":"2024-03-24T11:26:50.165830+0000","last_peered":"2024-03-24T11:26:50.165830+0000","last_clean":"2024-03-24T11:26:50.165830+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:26:50.165830+0000","last_undegraded":"2024-03-24T11:26:50.165830+0000","last_fullsized":"2024-03-24T11:26:50.165830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:42.826873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.826873+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:04.232390+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164223+0000","last_change":"2024-03-24T11:25:22.291763+0000","last_active":"2024-03-24T11:26:50.164223+0000","last_peered":"2024-03-24T11:26:50.164223+0000","last_clean":"2024-03-24T11:26:50.164223+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:26:50.164223+0000","last_undegraded":"2024-03-24T11:26:50.164223+0000","last_fullsized":"2024-03-24T11:26:50.164223+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:22.291490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:25:22.291490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"89'22","reported_seq":308,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.174840+0000","last_change":"2024-03-24T11:26:46.757168+0000","last_active":"2024-03-24T11:26:50.174840+0000","last_peered":"2024-03-24T11:26:50.174840+0000","last_clean":"2024-03-24T11:26:50.174840+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:26:50.174840+0000","last_undegraded":"2024-03-24T11:26:50.174840+0000","last_fullsized":"2024-03-24T11:26:50.174840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-03-24T11:26:46.757070+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.757070+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.851634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"179'8846","reported_seq":9000,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.108249+0000","last_change":"2024-03-24T11:26:48.243638+0000","last_active":"2024-03-24T11:26:58.108249+0000","last_peered":"2024-03-24T11:26:58.108249+0000","last_clean":"2024-03-24T11:26:58.108249+0000","last_became_active":"2024-03-24T11:25:02.599329+0000","last_became_peered":"2024-03-24T11:25:02.599329+0000","last_unstale":"2024-03-24T11:26:58.108249+0000","last_undegraded":"2024-03-24T11:26:58.108249+0000","last_fullsized":"2024-03-24T11:26:58.108249+0000","mapping_epoch":150,"log_start":"168'4300","ondisk_log_start":"168'4300","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8117","last_scrub_stamp":"2024-03-24T11:26:48.243210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.243210+0000","objects_scrubbed":7656,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.207893+0000","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450494464,"num_objects":6874,"num_object_clones":0,"num_object_copies":20622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":244,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.163852+0000","last_change":"2024-03-24T11:23:09.925823+0000","last_active":"2024-03-24T11:26:50.163852+0000","last_peered":"2024-03-24T11:26:50.163852+0000","last_clean":"2024-03-24T11:26:50.163852+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:26:50.163852+0000","last_undegraded":"2024-03-24T11:26:50.163852+0000","last_fullsized":"2024-03-24T11:26:50.163852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:09.925715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:09.925715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149124+0000","last_change":"2024-03-24T11:26:28.105930+0000","last_active":"2024-03-24T11:26:50.149124+0000","last_peered":"2024-03-24T11:26:50.149124+0000","last_clean":"2024-03-24T11:26:50.149124+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:26:50.149124+0000","last_undegraded":"2024-03-24T11:26:50.149124+0000","last_fullsized":"2024-03-24T11:26:50.149124+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.105784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:24:43.042611+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.105784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:48.780495+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.164939+0000","last_change":"2024-03-24T11:26:49.883140+0000","last_active":"2024-03-24T11:26:50.164939+0000","last_peered":"2024-03-24T11:26:50.164939+0000","last_clean":"2024-03-24T11:26:50.164939+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:26:50.164939+0000","last_undegraded":"2024-03-24T11:26:50.164939+0000","last_fullsized":"2024-03-24T11:26:50.164939+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.883061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.883061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:52.393144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":287,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165434+0000","last_change":"2024-03-24T11:25:43.030583+0000","last_active":"2024-03-24T11:26:50.165434+0000","last_peered":"2024-03-24T11:26:50.165434+0000","last_clean":"2024-03-24T11:26:50.165434+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:26:50.165434+0000","last_undegraded":"2024-03-24T11:26:50.165434+0000","last_fullsized":"2024-03-24T11:26:50.165434+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:25:43.030417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.030417+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"179'9045","reported_seq":9197,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.426418+0000","last_change":"2024-03-24T11:26:36.863675+0000","last_active":"2024-03-24T11:26:58.426418+0000","last_peered":"2024-03-24T11:26:58.426418+0000","last_clean":"2024-03-24T11:26:58.426418+0000","last_became_active":"2024-03-24T11:25:02.674932+0000","last_became_peered":"2024-03-24T11:25:02.674932+0000","last_unstale":"2024-03-24T11:26:58.426418+0000","last_undegraded":"2024-03-24T11:26:58.426418+0000","last_fullsized":"2024-03-24T11:26:58.426418+0000","mapping_epoch":150,"log_start":"167'4200","ondisk_log_start":"167'4200","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7579","last_scrub_stamp":"2024-03-24T11:26:36.863361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.863361+0000","objects_scrubbed":7488,"log_size":4845,"log_dups_size":0,"ondisk_log_size":4845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:06.565935+0000","scrub_duration":3086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458555392,"num_objects":6997,"num_object_clones":0,"num_object_copies":20991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9045,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":403,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.151740+0000","last_change":"2024-03-24T11:25:43.004840+0000","last_active":"2024-03-24T11:26:50.151740+0000","last_peered":"2024-03-24T11:26:50.151740+0000","last_clean":"2024-03-24T11:26:50.151740+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:26:50.151740+0000","last_undegraded":"2024-03-24T11:26:50.151740+0000","last_fullsized":"2024-03-24T11:26:50.151740+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:25:43.004596+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:25:43.004596+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.004596+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.170511+0000","last_change":"2024-03-24T11:25:25.841724+0000","last_active":"2024-03-24T11:26:50.170511+0000","last_peered":"2024-03-24T11:26:50.170511+0000","last_clean":"2024-03-24T11:26:50.170511+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:26:50.170511+0000","last_undegraded":"2024-03-24T11:26:50.170511+0000","last_fullsized":"2024-03-24T11:26:50.170511+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:25.841476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:25.841476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:51.820445+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165707+0000","last_change":"2024-03-24T11:26:48.793731+0000","last_active":"2024-03-24T11:26:50.165707+0000","last_peered":"2024-03-24T11:26:50.165707+0000","last_clean":"2024-03-24T11:26:50.165707+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:26:50.165707+0000","last_undegraded":"2024-03-24T11:26:50.165707+0000","last_fullsized":"2024-03-24T11:26:50.165707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.793614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.316226+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":286,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.149862+0000","last_change":"2024-03-24T11:25:43.745091+0000","last_active":"2024-03-24T11:26:50.149862+0000","last_peered":"2024-03-24T11:26:50.149862+0000","last_clean":"2024-03-24T11:26:50.149862+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:26:50.149862+0000","last_undegraded":"2024-03-24T11:26:50.149862+0000","last_fullsized":"2024-03-24T11:26:50.149862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:25:43.744958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.744958+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:26:56.754970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"88'25","reported_seq":328,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.167065+0000","last_change":"2024-03-24T11:25:49.109903+0000","last_active":"2024-03-24T11:26:50.167065+0000","last_peered":"2024-03-24T11:26:50.167065+0000","last_clean":"2024-03-24T11:26:50.167065+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:26:50.167065+0000","last_undegraded":"2024-03-24T11:26:50.167065+0000","last_fullsized":"2024-03-24T11:26:50.167065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'25","last_scrub_stamp":"2024-03-24T11:25:49.109688+0000","last_deep_scrub":"88'25","last_deep_scrub_stamp":"2024-03-24T11:25:49.109688+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.109688+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:16.118882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"179'9074","reported_seq":9142,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:58.609989+0000","last_change":"2024-03-24T11:25:02.599491+0000","last_active":"2024-03-24T11:26:58.609989+0000","last_peered":"2024-03-24T11:26:58.609989+0000","last_clean":"2024-03-24T11:26:58.609989+0000","last_became_active":"2024-03-24T11:25:02.599177+0000","last_became_peered":"2024-03-24T11:25:02.599177+0000","last_unstale":"2024-03-24T11:26:58.609989+0000","last_undegraded":"2024-03-24T11:26:58.609989+0000","last_fullsized":"2024-03-24T11:26:58.609989+0000","mapping_epoch":150,"log_start":"168'4500","ondisk_log_start":"168'4500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:03.387098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455081984,"num_objects":6944,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":512576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":542,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.152810+0000","last_change":"2024-03-24T11:25:47.305585+0000","last_active":"2024-03-24T11:26:50.152810+0000","last_peered":"2024-03-24T11:26:50.152810+0000","last_clean":"2024-03-24T11:26:50.152810+0000","last_became_active":"2024-03-24T11:24:29.091858+0000","last_became_peered":"2024-03-24T11:24:29.091858+0000","last_unstale":"2024-03-24T11:26:50.152810+0000","last_undegraded":"2024-03-24T11:26:50.152810+0000","last_fullsized":"2024-03-24T11:26:50.152810+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:47.305457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:47.305457+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:11.450997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165687+0000","last_change":"2024-03-24T11:26:37.859404+0000","last_active":"2024-03-24T11:26:50.165687+0000","last_peered":"2024-03-24T11:26:50.165687+0000","last_clean":"2024-03-24T11:26:50.165687+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:26:50.165687+0000","last_undegraded":"2024-03-24T11:26:50.165687+0000","last_fullsized":"2024-03-24T11:26:50.165687+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.859286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.859286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:47.488541+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":377,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.173462+0000","last_change":"2024-03-24T11:25:48.800516+0000","last_active":"2024-03-24T11:26:50.173462+0000","last_peered":"2024-03-24T11:26:50.173462+0000","last_clean":"2024-03-24T11:26:50.173462+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:26:50.173462+0000","last_undegraded":"2024-03-24T11:26:50.173462+0000","last_fullsized":"2024-03-24T11:26:50.173462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:48.800253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:48.800253+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:01.987169+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.157586+0000","last_change":"2024-03-24T11:26:46.366056+0000","last_active":"2024-03-24T11:26:50.157586+0000","last_peered":"2024-03-24T11:26:50.157586+0000","last_clean":"2024-03-24T11:26:50.157586+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:26:50.157586+0000","last_undegraded":"2024-03-24T11:26:50.157586+0000","last_fullsized":"2024-03-24T11:26:50.157586+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.365877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.365877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.222226+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":275,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-24T11:26:50.165778+0000","last_change":"2024-03-24T11:25:41.084396+0000","last_active":"2024-03-24T11:26:50.165778+0000","last_peered":"2024-03-24T11:26:50.165778+0000","last_clean":"2024-03-24T11:26:50.165778+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:26:50.165778+0000","last_undegraded":"2024-03-24T11:26:50.165778+0000","last_fullsized":"2024-03-24T11:26:50.165778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:25:41.084261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:41.084261+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:00.823096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7325286424,"num_objects":111776,"num_object_clones":0,"num_object_copies":335328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144304,"num_write_kb":8194497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21993435136,"data_stored":21993422920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80604,"ondisk_log_size":80604,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":477,"num_read_kb":472,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":448,"ondisk_log_size":448,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070565,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2426456,"kb_used_data":2246736,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91318696,"statfs":{"total":95995035648,"available":93510344704,"internally_reserved":0,"allocated":2300657664,"data_stored":2300337885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":184023499},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364906,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2432968,"kb_used_data":2257408,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91312184,"statfs":{"total":95995035648,"available":93503676416,"internally_reserved":0,"allocated":2311585792,"data_stored":2311282594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179763654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574980,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3407600,"kb_used_data":3173736,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90337552,"statfs":{"total":95995035648,"available":92505653248,"internally_reserved":0,"allocated":3249905664,"data_stored":3249562343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738505,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2946524,"kb_used_data":2662676,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":90798628,"statfs":{"total":95995035648,"available":92977795072,"internally_reserved":0,"allocated":2726580224,"data_stored":2726256080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411455,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2944088,"kb_used_data":2696336,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":90801064,"statfs":{"total":95995035648,"available":92980289536,"internally_reserved":0,"allocated":2761048064,"data_stored":2760728345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2983344,"kb_used_data":2709096,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":90761808,"statfs":{"total":95995035648,"available":92940091392,"internally_reserved":0,"allocated":2774114304,"data_stored":2773770719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986292,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2442848,"kb_used_data":2258968,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91302304,"statfs":{"total":95995035648,"available":93493559296,"internally_reserved":0,"allocated":2313183232,"data_stored":2312855665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659247,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3800468,"kb_used_data":3576332,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":89944684,"statfs":{"total":95995035648,"available":92103356416,"internally_reserved":0,"allocated":3662163968,"data_stored":3661848611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2710437888,"data_stored":2710437888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3226468352,"data_stored":3226468352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2747273216,"data_stored":2747269144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2763587584,"data_stored":2763587584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2306867200,"data_stored":2306867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3653701632,"data_stored":3653697560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2290745344,"data_stored":2290745344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2294353920,"data_stored":2294349848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:27:02.675 INFO:tasks.thrashosds.thrasher:Setting 3.2 to [2, 0, 1, 5, 6, 7] 2024-03-24T11:27:02.675 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.2', '2', '0', '1', '5', '6', '7'] 2024-03-24T11:27:02.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 3.2 2 0 1 5 6 7 2024-03-24T11:27:02.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:02.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:02.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:02.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:02.946 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:02.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:02.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:03.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:03.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:03.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:03.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:03.227 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:03.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:03.291 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:03 smithi181 ceph-mon[33998]: pgmap v324: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 970 op/s 2024-03-24T11:27:03.291 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:03 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:03.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:03 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:03.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:03 smithi181 ceph-mon[33998]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:27:03.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:03 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:03.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:03.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:03.358 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:27:03.358 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:27:03.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:03.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:03.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:03.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:03.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[29608]: pgmap v324: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 970 op/s 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[29608]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[37907]: pgmap v324: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 970 op/s 2024-03-24T11:27:03.528 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:03.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:03.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[37907]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T11:27:03.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:03.529 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:03.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:03.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:27:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:27:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:03.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:03.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:03.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:03.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:03.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:03.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:03.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:03.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:04.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:04.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:04.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:04.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T11:27:04.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:04.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:04.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:04.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:04.371 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:04.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:04.446 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:04.446 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:04.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:04.487 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:27:04.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:04.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:04.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:04.635 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:04.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:04.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:04.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:04.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:04.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:04.862 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:04.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:04.923 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:04.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:04.969 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:27:05.008 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:27:05.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:05.049 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:27:05.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:05.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:05.097 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:27:05.134 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:27:05.157 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:05.174 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:27:05.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:05.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:05.213 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:27:05.231 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:27:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:27:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:27:05.255 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:27:05.263 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:27:05.263 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:27:05.264 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:27:05.264 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:27:05.264 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:27:05.264 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:27:05.264 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:27:05.264 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:27:05.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:05.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:05.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:05.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:05.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:05.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:05.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:27:05.509 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:05 smithi179 ceph-mon[29608]: pgmap v326: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T11:27:05.509 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:05 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2013151519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:05.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:05 smithi179 ceph-mon[37907]: pgmap v326: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T11:27:05.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:05 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2013151519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:05.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:05.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:05.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:05.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:05 smithi181 ceph-mon[33998]: pgmap v326: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T11:27:05.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:05 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2013151519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:05.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:05.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:05.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:05.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:05.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:05.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:05.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:06.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:06.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:06.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:06.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:06.282 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:06.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:06.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:06.443 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:27:06.443 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:27:06.443 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-03-24T11:27:06.443 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:27:06.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.560 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:06.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:06.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:06 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:27:06.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:06 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:27:06.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:06 smithi181 ceph-mon[33998]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:27:06.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.702 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:06.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:06 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:27:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:06 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:27:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:06 smithi179 ceph-mon[29608]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:27:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:06 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:27:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:06 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:27:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:06 smithi179 ceph-mon[37907]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T11:27:06.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:06.832 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:06.934 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:06.975 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:06.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:07.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:07.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:07.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:07.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T11:27:07.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:07.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:07.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:07.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:07.390 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:07.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:07.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:07 smithi181 ceph-mon[33998]: pgmap v328: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:27:07.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:07 smithi179 ceph-mon[29608]: pgmap v328: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:27:07.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:07 smithi179 ceph-mon[37907]: pgmap v328: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.32k op/s 2024-03-24T11:27:07.492 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:07.527 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:07.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:07.545 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:27:07.549 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:07.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:07.687 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:07.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:07.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:07.833 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:07.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:07.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:07.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:07.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:08.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:08.116 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:08.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:08.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:08.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:08.267 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:08.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:08.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:08.406 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:08.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:08.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:08.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:08.540 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:08.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:08.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:08.673 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:08.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:08.774 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:08.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:08.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:08.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:08.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:08.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:09.057 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:09.103 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:09.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:09.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:09.236 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:09.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:09.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:09.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:09.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:09.435 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:27:09.435 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:27:09.435 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:27:09.435 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:27:09.435 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:27:09.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:09.503 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:09 smithi181 ceph-mon[33998]: pgmap v329: 158 pgs: 158 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s 2024-03-24T11:27:09.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:09.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:09.522 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:09 smithi179 ceph-mon[29608]: pgmap v329: 158 pgs: 158 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s 2024-03-24T11:27:09.522 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:09 smithi179 ceph-mon[37907]: pgmap v329: 158 pgs: 158 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s 2024-03-24T11:27:09.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:27:09.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:09.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:09.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:09.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:09.813 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:09.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:09.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:09.965 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:09.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:10.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:10.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:10.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:10.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T11:27:10.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:10.248 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:10.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:10.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:10.408 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:10.409 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:10.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:10.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:10.525 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:27:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:10.561 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:10.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:10.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:10.703 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:10.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:10.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:10.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:10.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:10.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:10.979 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:10.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:11.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:11.113 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:11.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:11.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:11.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:11.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:11.333 INFO:teuthology.orchestra.run.smithi179.stderr:set 3.2 pg_upmap_items mapping to [2->0,1->5,6->7] 2024-03-24T11:27:11.335 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:27:11.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:11.389 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:11.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:11.490 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:11.530 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:11.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:11.535 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[29608]: pgmap v330: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T11:27:11.535 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2196659224' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]: dispatch 2024-03-24T11:27:11.535 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]: dispatch 2024-03-24T11:27:11.535 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1833396857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:27:11.536 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:27:11.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[37907]: pgmap v330: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T11:27:11.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2196659224' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]: dispatch 2024-03-24T11:27:11.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]: dispatch 2024-03-24T11:27:11.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1833396857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:27:11.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:11 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:27:11.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:11.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:11 smithi181 ceph-mon[33998]: pgmap v330: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T11:27:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:11 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2196659224' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]: dispatch 2024-03-24T11:27:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:11 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]: dispatch 2024-03-24T11:27:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:11 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1833396857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:27:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:11 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:27:11.672 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:11.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:11.774 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:11.815 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:11.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:11.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:11.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:11.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:12.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:12.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:12.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:12.203 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:12.235 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:12.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:12.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:12.370 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:12.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:12.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:12 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:27:12.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:12 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:27:12.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:12 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:12.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:12 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]': finished 2024-03-24T11:27:12.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:12 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:27:12.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:12 smithi181 ceph-mon[33998]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:27:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]': finished 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[29608]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.2", "id": [2, 0, 1, 5, 6, 7]}]': finished 2024-03-24T11:27:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:27:12.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:12 smithi179 ceph-mon[37907]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T11:27:12.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:12.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:12.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:12.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:12.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:12.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:12.663 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:27:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:27:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:27:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:21748 7.5 7:a6556fc2:::benchmark_data_smithi179_78267_object21747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:15.731437+0000", 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 356.92961391699998, 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068710555000000006, 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 21748 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731437+0000", 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731437+0000", 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731438+0000", 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1990000000000001e-06 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731480+0000", 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1182e-05 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731490+0000", 2024-03-24T11:27:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0695e-05 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731498+0000", 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0620000000000001e-06 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731554+0000", 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5297000000000003e-05 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731680+0000", 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126379 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.731832+0000", 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151576 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.732818+0000", 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00098657699999999998 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.732939+0000", 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012116399999999999 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.800037+0000", 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067097956 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.800094+0000", 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7043999999999998e-05 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:15.800148+0000", 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3424000000000002e-05 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26050 7.5 7:a02b24c6:::benchmark_data_smithi179_78267_object26049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 354.841657699, 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109015764, 2024-03-24T11:27:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26050 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819394+0000", 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2999999999999999e-07 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819439+0000", 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4925000000000001e-05 2024-03-24T11:27:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819450+0000", 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0586e-05 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819455+0000", 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7219999999999996e-06 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819502+0000", 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6295000000000003e-05 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819583+0000", 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0733000000000001e-05 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819680+0000", 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7677999999999996e-05 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820508+0000", 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000827445 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.821885+0000", 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001377365 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928313+0000", 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106428435 2024-03-24T11:27:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928370+0000", 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6368e-05 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928409+0000", 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9381999999999997e-05 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26055 7.5 7:a6f221c1:::benchmark_data_smithi179_78267_object26054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 354.84036611200003, 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10789344200000001, 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26055 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820686+0000", 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2999999999999999e-07 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820747+0000", 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1148999999999993e-05 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820760+0000", 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820768+0000", 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7610000000000007e-06 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820814+0000", 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6644999999999998e-05 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820932+0000", 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117788 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.821060+0000", 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012757900000000001 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.822115+0000", 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010555689999999999 2024-03-24T11:27:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.822131+0000", 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5673000000000001e-05 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928469+0000", 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10633761 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928531+0000", 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2549e-05 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928578+0000", 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7308000000000002e-05 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26494 7.5 7:a4c9ac9c:::benchmark_data_smithi179_78267_object26493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:27:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 354.53778644599998, 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15363169800000001, 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26494 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123266+0000", 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1710000000000001e-06 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123350+0000", 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3776000000000006e-05 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123366+0000", 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6779999999999999e-05 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123375+0000", 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.439e-06 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123428+0000", 2024-03-24T11:27:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2760000000000003e-05 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123535+0000", 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107574 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123703+0000", 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016783 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.124976+0000", 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012726930000000001 2024-03-24T11:27:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125304+0000", 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032807300000000001 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276785+0000", 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.151481123 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276851+0000", 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6204999999999994e-05 2024-03-24T11:27:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276896+0000", 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5274000000000001e-05 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26500 7.5 7:afbe2af8:::benchmark_data_smithi179_78267_object26499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 354.53527483400001, 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15542166600000001, 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26500 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125777+0000", 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7100000000000011e-07 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125824+0000", 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6388999999999999e-05 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125835+0000", 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1032999999999999e-05 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125840+0000", 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6099999999999997e-06 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125886+0000", 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5711000000000003e-05 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125998+0000", 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111952 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126130+0000", 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013231699999999999 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127017+0000", 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00088662500000000004 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127330+0000", 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000313006 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281101+0000", 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15377119 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281159+0000", 2024-03-24T11:27:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7759e-05 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281198+0000", 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9103000000000002e-05 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:27107 7.5 7:a4201e14:::benchmark_data_smithi179_78267_object27106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.543620+0000", 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 354.11743135900002, 2024-03-24T11:27:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067854379000000006, 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 27107 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543620+0000", 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543620+0000", 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543621+0000", 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8100000000000001e-07 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543688+0000", 2024-03-24T11:27:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7092000000000001e-05 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543701+0000", 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2741e-05 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543709+0000", 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543778+0000", 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9062999999999999e-05 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.543902+0000", 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012383000000000001 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.544060+0000", 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015877299999999999 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.545099+0000", 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001038773 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.547163+0000", 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020637400000000001 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.611359+0000", 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.064196290000000003 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.611426+0000", 2024-03-24T11:27:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7287999999999997e-05 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.611474+0000", 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.774e-05 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:31022 7.5 7:a6087dad:::benchmark_data_smithi179_78267_object31021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:20.812672+0000", 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 351.848378615, 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078745411000000001, 2024-03-24T11:27:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 31022 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812672+0000", 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812672+0000", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812674+0000", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.037e-06 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812719+0000", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5053999999999999e-05 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812744+0000", 2024-03-24T11:27:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5018999999999999e-05 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812752+0000", 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4449999999999996e-06 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812818+0000", 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5874e-05 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.812927+0000", 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010874600000000001 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.813106+0000", 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000179459 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.814261+0000", 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011543949999999999 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.818357+0000", 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004096939 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.891317+0000", 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072959780000000002 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.891369+0000", 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1279000000000002e-05 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:20.891418+0000", 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9384e-05 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:64062 7.5 7:a7020b65:::benchmark_data_smithi179_78267_object64061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:42.573650+0000", 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 330.08740119700002, 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078293327999999995, 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 64062 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573650+0000", 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573650+0000", 2024-03-24T11:27:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573651+0000", 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.251e-06 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573696+0000", 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5191999999999997e-05 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573709+0000", 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.242e-05 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573718+0000", 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9360000000000005e-06 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573786+0000", 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8387999999999996e-05 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573886+0000", 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9471999999999999e-05 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.573989+0000", 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010310300000000001 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.576159+0000", 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021701720000000002 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.577205+0000", 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010460350000000001 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.651809+0000", 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074603931999999998 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.651881+0000", 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1705000000000005e-05 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:42.651943+0000", 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2722e-05 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:66218 7.5 7:a1d02a05:::benchmark_data_smithi179_78267_object66217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:27:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 328.03665878800001, 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087797980999999997, 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 66218 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624393+0000", 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0869999999999999e-06 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624444+0000", 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0929e-05 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624454+0000", 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4730000000000006e-06 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624460+0000", 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7180000000000001e-06 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624515+0000", 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5519999999999997e-05 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624671+0000", 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015631299999999999 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624776+0000", 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104378 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.628751+0000", 2024-03-24T11:27:12.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039755060000000002 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.686471+0000", 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057719422999999999 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712076+0000", 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025605297999999999 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712140+0000", 2024-03-24T11:27:12.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4049999999999998e-05 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712190+0000", 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0285999999999999e-05 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:66219 7.5 7:a6ed2970:::benchmark_data_smithi179_78267_object66218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 328.03656614599998, 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087858568999999997, 2024-03-24T11:27:12.720 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 66219 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624486+0000", 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8199999999999999e-07 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624563+0000", 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6958000000000004e-05 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624571+0000", 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3319999999999992e-06 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624577+0000", 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1619999999999996e-06 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626484+0000", 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019068620000000001 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626623+0000", 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013903099999999999 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626786+0000", 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000163096 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.628876+0000", 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020895620000000001 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.686625+0000", 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057748762000000002 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712252+0000", 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025627644000000002 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712307+0000", 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4877000000000003e-05 2024-03-24T11:27:12.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712344+0000", 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6501000000000002e-05 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:68243 7.5 7:a17adc7b:::benchmark_data_smithi179_78267_object68242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:46.516297+0000", 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.14475381, 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067671549999999997, 2024-03-24T11:27:12.726 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 68243 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516297+0000", 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516297+0000", 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516299+0000", 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6959999999999999e-06 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516353+0000", 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4018999999999999e-05 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516368+0000", 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5175e-05 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516377+0000", 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3450000000000006e-06 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516444+0000", 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7347000000000004e-05 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516614+0000", 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017001700000000001 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.516801+0000", 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018701300000000001 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521912+0000", 2024-03-24T11:27:12.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051114369999999999 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521980+0000", 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7380999999999997e-05 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.583889+0000", 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.061909750999999999 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.583938+0000", 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8121000000000002e-05 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.583969+0000", 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1248000000000003e-05 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:68248 7.5 7:a886aae0:::benchmark_data_smithi179_78267_object68247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:46.517950+0000", 2024-03-24T11:27:12.732 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.14310080500002, 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066190224000000006, 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 68248 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517950+0000", 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517950+0000", 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517952+0000", 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6500000000000001e-06 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517982+0000", 2024-03-24T11:27:12.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9611000000000002e-05 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.517997+0000", 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5104e-05 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.518005+0000", 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5339999999999999e-06 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.519133+0000", 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001127798 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.519293+0000", 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015965599999999999 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.519505+0000", 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000212015 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521971+0000", 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024665149999999999 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521989+0000", 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7584e-05 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.584016+0000", 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.062027269000000003 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.584097+0000", 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1283000000000001e-05 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.584141+0000", 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3204999999999997e-05 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.738 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:68251 7.5 7:ab028aee:::benchmark_data_smithi179_78267_object68250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:46.521793+0000", 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.13925859800003, 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.070321116000000003, 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 68251 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521793+0000", 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521793+0000", 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521794+0000", 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.189e-06 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521865+0000", 2024-03-24T11:27:12.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1311999999999998e-05 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521881+0000", 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6220000000000001e-05 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.521898+0000", 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.683e-05 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.522012+0000", 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011431500000000001 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.522162+0000", 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014995399999999999 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.522334+0000", 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171767 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.524633+0000", 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022989410000000001 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.524759+0000", 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126107 2024-03-24T11:27:12.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.591882+0000", 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.067122845 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.591939+0000", 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7432000000000002e-05 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:46.592114+0000", 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017420399999999999 2024-03-24T11:27:12.744 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:70839 7.5 7:a954ad76:::benchmark_data_smithi179_78267_object70838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:48.814138+0000", 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 323.84691313299999, 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074097784999999999, 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.745 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 70839 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814138+0000", 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814138+0000", 2024-03-24T11:27:12.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814140+0000", 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0679999999999999e-06 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814189+0000", 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8749999999999999e-05 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814204+0000", 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5238e-05 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814214+0000", 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0028e-05 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814290+0000", 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6008999999999997e-05 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814451+0000", 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160877 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.814607+0000", 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015562200000000001 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.817243+0000", 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026363760000000002 2024-03-24T11:27:12.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.817385+0000", 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014165299999999999 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.888075+0000", 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.070690826999999998 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.888176+0000", 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000101055 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:48.888236+0000", 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9281999999999999e-05 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:75763 7.5 7:a2642e9d:::benchmark_data_smithi179_78267_object75762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:53.173290+0000", 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 319.48776100499998, 2024-03-24T11:27:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072918858000000003, 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 75763 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173290+0000", 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173290+0000", 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173292+0000", 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.395e-06 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173346+0000", 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4299000000000003e-05 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173360+0000", 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4133e-05 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173366+0000", 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9839999999999998e-06 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173441+0000", 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5160000000000005e-05 2024-03-24T11:27:12.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173556+0000", 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114734 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.173659+0000", 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010362 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.176081+0000", 2024-03-24T11:27:12.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024217409999999998 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.177078+0000", 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00099639200000000002 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.246093+0000", 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.069015100999999995 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.246157+0000", 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4659999999999994e-05 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:53.246209+0000", 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1638999999999998e-05 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:92829 7.5 7:a6a4baa4:::benchmark_data_smithi179_78267_object92828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:09.778359+0000", 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 302.88269218599999, 2024-03-24T11:27:12.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.077760105999999996, 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 92829 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778359+0000", 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778359+0000", 2024-03-24T11:27:12.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778361+0000", 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.669e-06 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778418+0000", 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7343999999999998e-05 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778435+0000", 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7329999999999998e-05 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778444+0000", 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7339999999999997e-06 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778510+0000", 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6383999999999993e-05 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778661+0000", 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000150894 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.778822+0000", 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160421 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.782048+0000", 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003226745 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.783212+0000", 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011637469999999999 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.856010+0000", 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072798013999999994 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.856075+0000", 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4653999999999995e-05 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.856119+0000", 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4169999999999999e-05 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:92981 7.5 7:aa6b41c2:::benchmark_data_smithi179_78267_object92980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:09.961306+0000", 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 302.69974548900001, 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082104115000000005, 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 92981 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961306+0000", 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961306+0000", 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961307+0000", 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9399999999999993e-07 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961420+0000", 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000113791 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961457+0000", 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7027999999999999e-05 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.961469+0000", 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1399999999999999e-05 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.962674+0000", 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001205649 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.962829+0000", 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000154368 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.962986+0000", 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015717899999999999 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.965368+0000", 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023822230000000002 2024-03-24T11:27:12.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:09.966487+0000", 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011184700000000001 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:10.043332+0000", 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.076844777000000003 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:10.043374+0000", 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2055999999999998e-05 2024-03-24T11:27:12.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:10.043410+0000", 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6180000000000003e-05 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:154476 7.5 7:a41c9bf3:::benchmark_data_smithi179_78267_object35282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:15.098300+0000", 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 237.56275155200001, 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.065449867999999994, 2024-03-24T11:27:12.770 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 154476 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098300+0000", 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098300+0000", 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098301+0000", 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.316e-06 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098304+0000", 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4999999999999999e-06 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098316+0000", 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2038000000000001e-05 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098323+0000", 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9190000000000002e-06 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098380+0000", 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7105000000000003e-05 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098648+0000", 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026745099999999998 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.098772+0000", 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012398699999999999 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.102165+0000", 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033926730000000001 2024-03-24T11:27:12.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.107518+0000", 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053531539999999997 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.163637+0000", 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.056119166999999998 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.163704+0000", 2024-03-24T11:27:12.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6909000000000006e-05 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.163749+0000", 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5649000000000003e-05 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:177111 7.5 7:ad9fdd86:::benchmark_data_smithi179_78267_object57917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:38.969858+0000", 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 213.69119360799999, 2024-03-24T11:27:12.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066590238999999996, 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 177111 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969858+0000", 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969858+0000", 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969859+0000", 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5060000000000001e-06 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969865+0000", 2024-03-24T11:27:12.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7010000000000001e-06 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969880+0000", 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.573e-05 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969890+0000", 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8179999999999999e-06 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.969971+0000", 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0487000000000004e-05 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.970125+0000", 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015467899999999999 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.970278+0000", 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015304699999999999 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.973268+0000", 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029894129999999998 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:38.992670+0000", 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019401728 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:39.036295+0000", 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043625821000000002 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:39.036399+0000", 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103675 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:39.036448+0000", 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8634000000000002e-05 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.782 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:191750 7.5 7:a3905911:::benchmark_data_smithi179_78267_object72556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:53.650086+0000", 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 199.010965271, 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068069784999999994, 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 191750 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:27:12.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650086+0000", 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650086+0000", 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650088+0000", 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.815e-06 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650093+0000", 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3279999999999996e-06 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650106+0000", 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3339000000000001e-05 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650115+0000", 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2930000000000001e-06 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650183+0000", 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8658000000000006e-05 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650310+0000", 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012692500000000001 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.650419+0000", 2024-03-24T11:27:12.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108362 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.653111+0000", 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026921639999999999 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.714980+0000", 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.061868899999999998 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.718080+0000", 2024-03-24T11:27:12.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031008540000000001 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.718117+0000", 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6661e-05 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:53.718156+0000", 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8485999999999997e-05 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:27:12.788 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:27:12.789 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:27:12.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:12.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:12.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:12.931 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:12.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:12.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:13.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:13.111 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:13.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:13.213 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:13.256 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:13.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:13.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:13.383 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:13.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:13.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:13.524 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:13.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:13.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:13.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: pgmap v332: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 735 op/s 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:27:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:27:13.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:27:13.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:27:13.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:13 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:27:13.661 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:13.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:27:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: pgmap v332: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 735 op/s 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:27:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: pgmap v332: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 735 op/s 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:27:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:13 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:27:13.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:27:13.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:13.799 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:13.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:13.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:13.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:13.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:14.037 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:14.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:14.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:14.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:14.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:14.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:14.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:14.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:14.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:14.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:14.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:14.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:14.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:14.626 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:14.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:14.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:27:14.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:27:14.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:27:14.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:27:14.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:27:14.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: 4.10 scrub starts 2024-03-24T11:27:14.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: 4.10 scrub ok 2024-03-24T11:27:14.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:14.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:27:14.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:27:14.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:14 smithi181 ceph-mon[33998]: 8.8 scrub starts 2024-03-24T11:27:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:27:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: 4.10 scrub starts 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: 4.10 scrub ok 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[29608]: 8.8 scrub starts 2024-03-24T11:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: 4.10 scrub starts 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: 4.10 scrub ok 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:27:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:14 smithi179 ceph-mon[37907]: 8.8 scrub starts 2024-03-24T11:27:14.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:14.765 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:14.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:14.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:14.900 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:14.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:15.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.037 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:15.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:15.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:15.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:15.274 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:15.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:15.339 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:27:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:27:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:27:15.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.445 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:15.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:15.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:15.583 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:15.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:15 smithi181 ceph-mon[33998]: pgmap v335: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.94k op/s 2024-03-24T11:27:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:15 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:27:15.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:15 smithi179 ceph-mon[29608]: pgmap v335: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.94k op/s 2024-03-24T11:27:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:15 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:27:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:15 smithi179 ceph-mon[37907]: pgmap v335: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.94k op/s 2024-03-24T11:27:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:15 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:27:15.722 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:15.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.867 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:15.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:15.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:15.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:15.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:16.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:16.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:16.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:16.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:16.251 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:16.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:16.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:16.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:16.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:16.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:16.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:16.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:16.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:16 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:27:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:16 smithi181 ceph-mon[33998]: 8.6 scrub starts 2024-03-24T11:27:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:16 smithi181 ceph-mon[33998]: 8.8 scrub ok 2024-03-24T11:27:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:16.662 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:16.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:16.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:16 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:27:16.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:16 smithi179 ceph-mon[29608]: 8.6 scrub starts 2024-03-24T11:27:16.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:16 smithi179 ceph-mon[29608]: 8.8 scrub ok 2024-03-24T11:27:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:16 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:27:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:16 smithi179 ceph-mon[37907]: 8.6 scrub starts 2024-03-24T11:27:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:16 smithi179 ceph-mon[37907]: 8.8 scrub ok 2024-03-24T11:27:16.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:16.797 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:16.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:16.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:16.933 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:16.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:17.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:17.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:17.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:17.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:17.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:17.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:17.214 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:27:17.214 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:27:17.214 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-24T11:27:17.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:27:17.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:17.334 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:17.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:17.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:17.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:17.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:17.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:27:17.507 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:27:17.507 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: pgmap v336: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:27:17.507 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: 8.c scrub starts 2024-03-24T11:27:17.507 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:27:17.507 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:27:17.507 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:27:17.507 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:27:17.508 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:27:17.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:27:17.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: pgmap v336: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:27:17.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: 8.c scrub starts 2024-03-24T11:27:17.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:27:17.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:27:17.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:27:17.509 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:17 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:27:17.579 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:17.627 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:17.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:17.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:27:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:27:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: pgmap v336: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:27:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: 8.c scrub starts 2024-03-24T11:27:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:27:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:27:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:27:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:17 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:27:17.728 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:17.755 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:27:17.773 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:17.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:17.813 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:27:17.865 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:27:17.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:17.893 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:17.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:17.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:17.910 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:27:17.953 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:27:17.987 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:17.992 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:27:18.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:18.034 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:27:18.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:18.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:18.077 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:27:18.079 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:27:18.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:18.187 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:18.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:18.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:18.328 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:18.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:18.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:18.473 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:18.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:18.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:18.611 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:18.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:18 smithi181 ceph-mon[33998]: 8.e scrub starts 2024-03-24T11:27:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:18 smithi181 ceph-mon[33998]: 6.c scrub starts 2024-03-24T11:27:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:18 smithi181 ceph-mon[33998]: 6.c scrub ok 2024-03-24T11:27:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:18 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1639136671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:18.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[29608]: 8.e scrub starts 2024-03-24T11:27:18.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[29608]: 6.c scrub starts 2024-03-24T11:27:18.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[29608]: 6.c scrub ok 2024-03-24T11:27:18.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1639136671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:18.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[37907]: 8.e scrub starts 2024-03-24T11:27:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[37907]: 6.c scrub starts 2024-03-24T11:27:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[37907]: 6.c scrub ok 2024-03-24T11:27:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1639136671' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:18.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:18.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:18.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:18.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:18.878 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:18.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:18.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:19.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:19.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:19.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:19.161 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:19.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:19.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:19.292 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:19.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:19.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:19.436 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:19.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:19.505 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 6.1c scrub starts 2024-03-24T11:27:19.505 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 6.1c scrub ok 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: pgmap v337: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 929 op/s 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 6.1c scrub starts 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 6.1c scrub ok 2024-03-24T11:27:19.506 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:27:19.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:27:19.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: pgmap v337: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 929 op/s 2024-03-24T11:27:19.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:27:19.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:27:19.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:27:19.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:19 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:27:19.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:19.568 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:19.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 6.1c scrub starts 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 6.1c scrub ok 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: pgmap v337: 158 pgs: 158 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 929 op/s 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:27:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:19 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:27:19.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:19.710 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:19.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:19.812 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:19.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:27:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:19.854 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:19.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:19.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.011 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:20.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:20.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.151 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:20.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:20.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:20.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:20.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:20.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:20.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:20.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[29608]: 8.6 scrub ok 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[29608]: 8.c scrub ok 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[37907]: 8.6 scrub ok 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[37907]: 8.c scrub ok 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:27:20.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:20 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:27:20.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:20 smithi181 ceph-mon[33998]: 8.6 scrub ok 2024-03-24T11:27:20.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:20 smithi181 ceph-mon[33998]: 8.c scrub ok 2024-03-24T11:27:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:20 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:27:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:20 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:27:20.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:20.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:20.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.842 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:20.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:20.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:20.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:20.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:20.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:27:21.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:21.106 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:21.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:21.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:21.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:21.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:21.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:21.338 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:21.408 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:21.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[29608]: pgmap v338: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[29608]: 8.e scrub ok 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:27:21.444 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:27:21.445 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[37907]: pgmap v338: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-24T11:27:21.445 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:21 smithi179 ceph-mon[37907]: 8.e scrub ok 2024-03-24T11:27:21.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:21.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:21.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:21 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:27:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:21 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:27:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:21 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:27:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:21 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:27:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:21 smithi181 ceph-mon[33998]: pgmap v338: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-24T11:27:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:21 smithi181 ceph-mon[33998]: 8.e scrub ok 2024-03-24T11:27:21.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:21.706 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:21.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:21.721 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:21.810 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:21.856 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:21.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:21.865 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:27:21.869 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:27:21.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:21.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:21.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:22.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:22.132 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:22.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:22.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:22.271 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:22.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:22.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:22.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:22.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:22.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:22.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:22.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:22.520 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:22 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:27:22.520 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:22 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:27:22.520 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:22 smithi179 ceph-mon[29608]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:27:22.521 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:22 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:27:22.521 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:22 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:27:22.521 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:22 smithi179 ceph-mon[37907]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:27:22.538 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:22.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:22.640 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:22.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:22 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:27:22.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:22 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:27:22.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:22 smithi181 ceph-mon[33998]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:27:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:22.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:22.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:22.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:22.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:22.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:22.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:22.955 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:22.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:23.056 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:23.092 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:23.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:23.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:23.231 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:23.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:23.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:23.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:23.372 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:23.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:23.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:23.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:23.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:23.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:23.648 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:23.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:23.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:27:23.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:27:23.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-mon[33998]: pgmap v339: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:27:23.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:27:23.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:27:23.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:27:23.655 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:27:23.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:27:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:27:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:27:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:27:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[29608]: pgmap v339: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:27:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:27:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[37907]: pgmap v339: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:27:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:23 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:27:23.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:23.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:23.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:23.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:23.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:23.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:24.009 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:24.047 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:24.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:24.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:24.189 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:24.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:24.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:24.330 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:24.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:24.394 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:24.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:24.463 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:24.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:24.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:24.609 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:24.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:24 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:27:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:24 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:27:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:24 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:27:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:24 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:27:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:24 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:27:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:24 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:27:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:27:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:27:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:27:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:24 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:27:24.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:24.749 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:24.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:24.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:24.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:24.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:24.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:25.022 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:25.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:25.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:25.166 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:25.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:25.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:25.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:25.305 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:25.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:25.401 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:27:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:27:25.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:25.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:25.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:25.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:25.583 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:25.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:25 smithi181 ceph-mon[33998]: 6.2 scrub starts 2024-03-24T11:27:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:25 smithi181 ceph-mon[33998]: 6.2 scrub ok 2024-03-24T11:27:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:25 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:27:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:25 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:27:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:25 smithi181 ceph-mon[33998]: pgmap v340: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 944 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-24T11:27:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:25 smithi181 ceph-mon[33998]: 4.14 scrub starts 2024-03-24T11:27:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:25 smithi181 ceph-mon[33998]: 4.14 scrub ok 2024-03-24T11:27:25.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[29608]: 6.2 scrub starts 2024-03-24T11:27:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[29608]: 6.2 scrub ok 2024-03-24T11:27:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:27:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:27:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[29608]: pgmap v340: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 944 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[29608]: 4.14 scrub starts 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[29608]: 4.14 scrub ok 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[37907]: 6.2 scrub starts 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[37907]: 6.2 scrub ok 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[37907]: pgmap v340: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 944 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[37907]: 4.14 scrub starts 2024-03-24T11:27:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:25 smithi179 ceph-mon[37907]: 4.14 scrub ok 2024-03-24T11:27:25.728 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:25.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:25.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:25.874 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:25.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:25.956 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:27:25.956 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:27:25.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:26.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:26.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.146 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:26.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:26.247 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.288 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:26.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:26.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:26.429 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:26.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:26.554 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:27:26.554 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:27:26.554 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[29608]: 8.4 scrub starts 2024-03-24T11:27:26.554 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:27:26.554 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:27:26.555 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:27:26.555 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:27:26.555 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[37907]: 8.4 scrub starts 2024-03-24T11:27:26.555 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:27:26.555 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:26 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:27:26.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:26.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:26.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:26.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:26.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:26 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:27:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:26 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:27:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:26 smithi181 ceph-mon[33998]: 8.4 scrub starts 2024-03-24T11:27:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:26 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:27:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:26 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:27:26.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:26.969 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:26.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:27.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:27.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:27.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:27.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:27.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:27.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:27.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:27.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:27.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:27.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:27:27.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:27.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:27.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:27.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:27.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:27.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:27.642 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: pgmap v341: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 873 op/s 2024-03-24T11:27:27.642 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: 6.b scrub starts 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: 6.b scrub ok 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2876907147' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[29608]: 8.4 scrub ok 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: pgmap v341: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 873 op/s 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: 6.b scrub starts 2024-03-24T11:27:27.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: 6.b scrub ok 2024-03-24T11:27:27.644 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2876907147' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:27.644 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:27:27.644 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:27:27.644 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:27 smithi179 ceph-mon[37907]: 8.4 scrub ok 2024-03-24T11:27:27.699 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-03-24T11:27:20.176591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5818554742,"num_objects":88974,"num_object_clones":0,"num_object_copies":266922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88974,"num_whiteouts":0,"num_read":1219,"num_read_kb":971,"num_write":167740,"num_write_kb":8194542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82944,"ondisk_log_size":82944,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18987016,"kb_used_data":17146440,"kb_used_omap":12,"kb_used_meta":1840307,"kb_avail":730974200,"statfs":{"total":767960285184,"available":748517580800,"internally_reserved":0,"allocated":17557954560,"data_stored":17555526682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1884474929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-729743360,"num_objects":-11135,"num_object_clones":0,"num_object_copies":-33405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11135,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002070"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346026+0000","last_change":"2024-03-24T11:26:32.919289+0000","last_active":"2024-03-24T11:27:13.346026+0000","last_peered":"2024-03-24T11:27:13.346026+0000","last_clean":"2024-03-24T11:27:13.346026+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:27:13.346026+0000","last_undegraded":"2024-03-24T11:27:13.346026+0000","last_fullsized":"2024-03-24T11:27:13.346026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.919158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.919158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:46.842537+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:17.030184+0000","last_change":"2024-03-24T11:27:17.030232+0000","last_active":"2024-03-24T11:27:17.030184+0000","last_peered":"2024-03-24T11:27:17.030184+0000","last_clean":"2024-03-24T11:27:17.030184+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:27:17.030184+0000","last_undegraded":"2024-03-24T11:27:17.030184+0000","last_fullsized":"2024-03-24T11:27:17.030184+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.030061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.030061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:23.838953+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":275,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340595+0000","last_change":"2024-03-24T11:25:36.241293+0000","last_active":"2024-03-24T11:27:13.340595+0000","last_peered":"2024-03-24T11:27:13.340595+0000","last_clean":"2024-03-24T11:27:13.340595+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:27:13.340595+0000","last_undegraded":"2024-03-24T11:27:13.340595+0000","last_fullsized":"2024-03-24T11:27:13.340595+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:36.241166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.241166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":310,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346319+0000","last_change":"2024-03-24T11:26:40.754563+0000","last_active":"2024-03-24T11:27:13.346319+0000","last_peered":"2024-03-24T11:27:13.346319+0000","last_clean":"2024-03-24T11:27:13.346319+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:27:13.346319+0000","last_undegraded":"2024-03-24T11:27:13.346319+0000","last_fullsized":"2024-03-24T11:27:13.346319+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:26:40.754425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:40.754425+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:40.933020+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":254,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.386323+0000","last_change":"2024-03-24T11:25:38.404738+0000","last_active":"2024-03-24T11:27:13.386323+0000","last_peered":"2024-03-24T11:27:13.386323+0000","last_clean":"2024-03-24T11:27:13.386323+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:27:13.386323+0000","last_undegraded":"2024-03-24T11:27:13.386323+0000","last_fullsized":"2024-03-24T11:27:13.386323+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:38.404511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.404511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345080+0000","last_change":"2024-03-24T11:27:11.705436+0000","last_active":"2024-03-24T11:27:13.345080+0000","last_peered":"2024-03-24T11:27:13.345080+0000","last_clean":"2024-03-24T11:27:13.345080+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:27:13.345080+0000","last_undegraded":"2024-03-24T11:27:13.345080+0000","last_fullsized":"2024-03-24T11:27:13.345080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:11.705334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.705334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.113548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":267,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.386867+0000","last_change":"2024-03-24T11:23:44.804746+0000","last_active":"2024-03-24T11:27:13.386867+0000","last_peered":"2024-03-24T11:27:13.386867+0000","last_clean":"2024-03-24T11:27:13.386867+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:27:13.386867+0000","last_undegraded":"2024-03-24T11:27:13.386867+0000","last_fullsized":"2024-03-24T11:27:13.386867+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:44.804631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:23:44.804631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":287,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.385949+0000","last_change":"2024-03-24T11:26:29.288367+0000","last_active":"2024-03-24T11:27:13.385949+0000","last_peered":"2024-03-24T11:27:13.385949+0000","last_clean":"2024-03-24T11:27:13.385949+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:27:13.385949+0000","last_undegraded":"2024-03-24T11:27:13.385949+0000","last_fullsized":"2024-03-24T11:27:13.385949+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:26:29.288223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.288223+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:31.719337+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:14.464053+0000","last_change":"2024-03-24T11:27:14.464091+0000","last_active":"2024-03-24T11:27:14.464053+0000","last_peered":"2024-03-24T11:27:14.464053+0000","last_clean":"2024-03-24T11:27:14.464053+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:27:14.464053+0000","last_undegraded":"2024-03-24T11:27:14.464053+0000","last_fullsized":"2024-03-24T11:27:14.464053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:14.463973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:14.463973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:16.251800+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340531+0000","last_change":"2024-03-24T11:25:32.085917+0000","last_active":"2024-03-24T11:27:13.340531+0000","last_peered":"2024-03-24T11:27:13.340531+0000","last_clean":"2024-03-24T11:27:13.340531+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:27:13.340531+0000","last_undegraded":"2024-03-24T11:27:13.340531+0000","last_fullsized":"2024-03-24T11:27:13.340531+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:32.085822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:32.085822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341227+0000","last_change":"2024-03-24T11:25:46.067948+0000","last_active":"2024-03-24T11:27:13.341227+0000","last_peered":"2024-03-24T11:27:13.341227+0000","last_clean":"2024-03-24T11:27:13.341227+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:27:13.341227+0000","last_undegraded":"2024-03-24T11:27:13.341227+0000","last_fullsized":"2024-03-24T11:27:13.341227+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:25:46.067824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":286,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346261+0000","last_change":"2024-03-24T11:25:29.099118+0000","last_active":"2024-03-24T11:27:13.346261+0000","last_peered":"2024-03-24T11:27:13.346261+0000","last_clean":"2024-03-24T11:27:13.346261+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:27:13.346261+0000","last_undegraded":"2024-03-24T11:27:13.346261+0000","last_fullsized":"2024-03-24T11:27:13.346261+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:25:29.098999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:29.098999+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.721530+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.343297+0000","last_change":"2024-03-24T11:26:42.053183+0000","last_active":"2024-03-24T11:27:13.343297+0000","last_peered":"2024-03-24T11:27:13.343297+0000","last_clean":"2024-03-24T11:27:13.343297+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:27:13.343297+0000","last_undegraded":"2024-03-24T11:27:13.343297+0000","last_fullsized":"2024-03-24T11:27:13.343297+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.052777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.052777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:45.628691+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":266,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346027+0000","last_change":"2024-03-24T11:25:31.258073+0000","last_active":"2024-03-24T11:27:13.346027+0000","last_peered":"2024-03-24T11:27:13.346027+0000","last_clean":"2024-03-24T11:27:13.346027+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:27:13.346027+0000","last_undegraded":"2024-03-24T11:27:13.346027+0000","last_fullsized":"2024-03-24T11:27:13.346027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:31.257974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:25:31.257974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341113+0000","last_change":"2024-03-24T11:26:43.090579+0000","last_active":"2024-03-24T11:27:13.341113+0000","last_peered":"2024-03-24T11:27:13.341113+0000","last_clean":"2024-03-24T11:27:13.341113+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:27:13.341113+0000","last_undegraded":"2024-03-24T11:27:13.341113+0000","last_fullsized":"2024-03-24T11:27:13.341113+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:43.090367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.090367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:06.561184+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":288,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346538+0000","last_change":"2024-03-24T11:26:47.926021+0000","last_active":"2024-03-24T11:27:13.346538+0000","last_peered":"2024-03-24T11:27:13.346538+0000","last_clean":"2024-03-24T11:27:13.346538+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:27:13.346538+0000","last_undegraded":"2024-03-24T11:27:13.346538+0000","last_fullsized":"2024-03-24T11:27:13.346538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-24T11:26:47.925773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.925773+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:56.281392+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":271,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347427+0000","last_change":"2024-03-24T11:26:27.996936+0000","last_active":"2024-03-24T11:27:13.347427+0000","last_peered":"2024-03-24T11:27:13.347427+0000","last_clean":"2024-03-24T11:27:13.347427+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:27:13.347427+0000","last_undegraded":"2024-03-24T11:27:13.347427+0000","last_fullsized":"2024-03-24T11:27:13.347427+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:27.996815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.996815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:50.789628+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.078060+0000","last_change":"2024-03-24T11:27:18.078096+0000","last_active":"2024-03-24T11:27:18.078060+0000","last_peered":"2024-03-24T11:27:18.078060+0000","last_clean":"2024-03-24T11:27:18.078060+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:27:18.078060+0000","last_undegraded":"2024-03-24T11:27:18.078060+0000","last_fullsized":"2024-03-24T11:27:18.078060+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:18.077961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.077961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:38.436426+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340317+0000","last_change":"2024-03-24T11:26:32.451559+0000","last_active":"2024-03-24T11:27:13.340317+0000","last_peered":"2024-03-24T11:27:13.340317+0000","last_clean":"2024-03-24T11:27:13.340317+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:27:13.340317+0000","last_undegraded":"2024-03-24T11:27:13.340317+0000","last_fullsized":"2024-03-24T11:27:13.340317+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.451446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.451446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:39.602821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":303,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.377760+0000","last_change":"2024-03-24T11:25:39.721869+0000","last_active":"2024-03-24T11:27:13.377760+0000","last_peered":"2024-03-24T11:27:13.377760+0000","last_clean":"2024-03-24T11:27:13.377760+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:27:13.377760+0000","last_undegraded":"2024-03-24T11:27:13.377760+0000","last_fullsized":"2024-03-24T11:27:13.377760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:25:39.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.721719+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.984063+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":255,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.387288+0000","last_change":"2024-03-24T11:24:26.157701+0000","last_active":"2024-03-24T11:27:13.387288+0000","last_peered":"2024-03-24T11:27:13.387288+0000","last_clean":"2024-03-24T11:27:13.387288+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:27:13.387288+0000","last_undegraded":"2024-03-24T11:27:13.387288+0000","last_fullsized":"2024-03-24T11:27:13.387288+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:24:26.157583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:24:26.157583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345559+0000","last_change":"2024-03-24T11:26:26.941914+0000","last_active":"2024-03-24T11:27:13.345559+0000","last_peered":"2024-03-24T11:27:13.345559+0000","last_clean":"2024-03-24T11:27:13.345559+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:27:13.345559+0000","last_undegraded":"2024-03-24T11:27:13.345559+0000","last_fullsized":"2024-03-24T11:27:13.345559+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.941809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.941809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:33.817485+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346353+0000","last_change":"2024-03-24T11:25:30.083197+0000","last_active":"2024-03-24T11:27:13.346353+0000","last_peered":"2024-03-24T11:27:13.346353+0000","last_clean":"2024-03-24T11:27:13.346353+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:27:13.346353+0000","last_undegraded":"2024-03-24T11:27:13.346353+0000","last_fullsized":"2024-03-24T11:27:13.346353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:30.082634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:30.082634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":285,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.386515+0000","last_change":"2024-03-24T11:25:39.305828+0000","last_active":"2024-03-24T11:27:13.386515+0000","last_peered":"2024-03-24T11:27:13.386515+0000","last_clean":"2024-03-24T11:27:13.386515+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:27:13.386515+0000","last_undegraded":"2024-03-24T11:27:13.386515+0000","last_fullsized":"2024-03-24T11:27:13.386515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:25:39.305712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.305712+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":272,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:17.910106+0000","last_change":"2024-03-24T11:27:17.910132+0000","last_active":"2024-03-24T11:27:17.910106+0000","last_peered":"2024-03-24T11:27:17.910106+0000","last_clean":"2024-03-24T11:27:17.910106+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:27:17.910106+0000","last_undegraded":"2024-03-24T11:27:17.910106+0000","last_fullsized":"2024-03-24T11:27:17.910106+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:27:17.910050+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.910050+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:26.378109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":391,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.825906+0000","last_change":"2024-03-24T11:25:18.874358+0000","last_active":"2024-03-24T11:27:13.825906+0000","last_peered":"2024-03-24T11:27:13.825906+0000","last_clean":"2024-03-24T11:27:13.825906+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:27:13.825906+0000","last_undegraded":"2024-03-24T11:27:13.825906+0000","last_fullsized":"2024-03-24T11:27:13.825906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:25:18.874179+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:23:53.129429+0000","last_clean_scrub_stamp":"2024-03-24T11:25:18.874179+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":291,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.388369+0000","last_change":"2024-03-24T11:26:26.669460+0000","last_active":"2024-03-24T11:27:13.388369+0000","last_peered":"2024-03-24T11:27:13.388369+0000","last_clean":"2024-03-24T11:27:13.388369+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:27:13.388369+0000","last_undegraded":"2024-03-24T11:27:13.388369+0000","last_fullsized":"2024-03-24T11:27:13.388369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:26:26.669216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.669216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:47.472918+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":294,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340730+0000","last_change":"2024-03-24T11:25:35.265042+0000","last_active":"2024-03-24T11:27:13.340730+0000","last_peered":"2024-03-24T11:27:13.340730+0000","last_clean":"2024-03-24T11:27:13.340730+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:27:13.340730+0000","last_undegraded":"2024-03-24T11:27:13.340730+0000","last_fullsized":"2024-03-24T11:27:13.340730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:25:35.264757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.264757+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.383280+0000","last_change":"2024-03-24T11:25:46.053120+0000","last_active":"2024-03-24T11:27:13.383280+0000","last_peered":"2024-03-24T11:27:13.383280+0000","last_clean":"2024-03-24T11:27:13.383280+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:27:13.383280+0000","last_undegraded":"2024-03-24T11:27:13.383280+0000","last_fullsized":"2024-03-24T11:27:13.383280+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:46.052975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:58.788469+0000","last_clean_scrub_stamp":"2024-03-24T11:25:46.052975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":275,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340821+0000","last_change":"2024-03-24T11:26:31.491413+0000","last_active":"2024-03-24T11:27:13.340821+0000","last_peered":"2024-03-24T11:27:13.340821+0000","last_clean":"2024-03-24T11:27:13.340821+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:27:13.340821+0000","last_undegraded":"2024-03-24T11:27:13.340821+0000","last_fullsized":"2024-03-24T11:27:13.340821+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:31.491292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:31.491292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.469946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.389910+0000","last_change":"2024-03-24T11:26:26.240418+0000","last_active":"2024-03-24T11:27:13.389910+0000","last_peered":"2024-03-24T11:27:13.389910+0000","last_clean":"2024-03-24T11:27:13.389910+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:27:13.389910+0000","last_undegraded":"2024-03-24T11:27:13.389910+0000","last_fullsized":"2024-03-24T11:27:13.389910+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.240302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.240302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:38.755012+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":311,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341031+0000","last_change":"2024-03-24T11:25:45.772885+0000","last_active":"2024-03-24T11:27:13.341031+0000","last_peered":"2024-03-24T11:27:13.341031+0000","last_clean":"2024-03-24T11:27:13.341031+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:27:13.341031+0000","last_undegraded":"2024-03-24T11:27:13.341031+0000","last_fullsized":"2024-03-24T11:27:13.341031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:25:45.772734+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:25:45.772734+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.492075+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":265,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345810+0000","last_change":"2024-03-24T11:25:22.230485+0000","last_active":"2024-03-24T11:27:13.345810+0000","last_peered":"2024-03-24T11:27:13.345810+0000","last_clean":"2024-03-24T11:27:13.345810+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:27:13.345810+0000","last_undegraded":"2024-03-24T11:27:13.345810+0000","last_fullsized":"2024-03-24T11:27:13.345810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:22.230155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:22.230155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938305+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.342187+0000","last_change":"2024-03-24T11:26:47.560502+0000","last_active":"2024-03-24T11:27:13.342187+0000","last_peered":"2024-03-24T11:27:13.342187+0000","last_clean":"2024-03-24T11:27:13.342187+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:27:13.342187+0000","last_undegraded":"2024-03-24T11:27:13.342187+0000","last_fullsized":"2024-03-24T11:27:13.342187+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:47.560381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.560381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.212240+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340653+0000","last_change":"2024-03-24T11:26:28.556498+0000","last_active":"2024-03-24T11:27:13.340653+0000","last_peered":"2024-03-24T11:27:13.340653+0000","last_clean":"2024-03-24T11:27:13.340653+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:27:13.340653+0000","last_undegraded":"2024-03-24T11:27:13.340653+0000","last_fullsized":"2024-03-24T11:27:13.340653+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.556368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.556368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:33.125296+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"87'6","reported_seq":278,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.458413+0000","last_change":"2024-03-24T11:27:13.458460+0000","last_active":"2024-03-24T11:27:13.458413+0000","last_peered":"2024-03-24T11:27:13.458413+0000","last_clean":"2024-03-24T11:27:13.458413+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:27:13.458413+0000","last_undegraded":"2024-03-24T11:27:13.458413+0000","last_fullsized":"2024-03-24T11:27:13.458413+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'6","last_scrub_stamp":"2024-03-24T11:27:13.458330+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:27:13.458330+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.492165+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.386408+0000","last_change":"2024-03-24T11:24:46.271791+0000","last_active":"2024-03-24T11:27:13.386408+0000","last_peered":"2024-03-24T11:27:13.386408+0000","last_clean":"2024-03-24T11:27:13.386408+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:27:13.386408+0000","last_undegraded":"2024-03-24T11:27:13.386408+0000","last_fullsized":"2024-03-24T11:27:13.386408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:24:46.271695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:24:46.271695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":266,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347007+0000","last_change":"2024-03-24T11:25:38.219467+0000","last_active":"2024-03-24T11:27:13.347007+0000","last_peered":"2024-03-24T11:27:13.347007+0000","last_clean":"2024-03-24T11:27:13.347007+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:27:13.347007+0000","last_undegraded":"2024-03-24T11:27:13.347007+0000","last_fullsized":"2024-03-24T11:27:13.347007+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:38.219347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.219347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341296+0000","last_change":"2024-03-24T11:26:30.097875+0000","last_active":"2024-03-24T11:27:13.341296+0000","last_peered":"2024-03-24T11:27:13.341296+0000","last_clean":"2024-03-24T11:27:13.341296+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:27:13.341296+0000","last_undegraded":"2024-03-24T11:27:13.341296+0000","last_fullsized":"2024-03-24T11:27:13.341296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:30.097693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:30.097693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:32.045056+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":281,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.385705+0000","last_change":"2024-03-24T11:24:29.091627+0000","last_active":"2024-03-24T11:27:13.385705+0000","last_peered":"2024-03-24T11:27:13.385705+0000","last_clean":"2024-03-24T11:27:13.385705+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:27:13.385705+0000","last_undegraded":"2024-03-24T11:27:13.385705+0000","last_fullsized":"2024-03-24T11:27:13.385705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:24:29.091502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:24:29.091502+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.339910+0000","last_change":"2024-03-24T11:25:27.393223+0000","last_active":"2024-03-24T11:27:13.339910+0000","last_peered":"2024-03-24T11:27:13.339910+0000","last_clean":"2024-03-24T11:27:13.339910+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:27:13.339910+0000","last_undegraded":"2024-03-24T11:27:13.339910+0000","last_fullsized":"2024-03-24T11:27:13.339910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:27.393103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:27.393103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":266,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.386283+0000","last_change":"2024-03-24T11:27:12.419817+0000","last_active":"2024-03-24T11:27:13.386283+0000","last_peered":"2024-03-24T11:27:13.386283+0000","last_clean":"2024-03-24T11:27:13.386283+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:27:13.386283+0000","last_undegraded":"2024-03-24T11:27:13.386283+0000","last_fullsized":"2024-03-24T11:27:13.386283+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:12.419688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:12.419688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.788280+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347512+0000","last_change":"2024-03-24T11:25:37.212084+0000","last_active":"2024-03-24T11:27:13.347512+0000","last_peered":"2024-03-24T11:27:13.347512+0000","last_clean":"2024-03-24T11:27:13.347512+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:27:13.347512+0000","last_undegraded":"2024-03-24T11:27:13.347512+0000","last_fullsized":"2024-03-24T11:27:13.347512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:37.212008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:37.212008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":294,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.382571+0000","last_change":"2024-03-24T11:25:36.929044+0000","last_active":"2024-03-24T11:27:13.382571+0000","last_peered":"2024-03-24T11:27:13.382571+0000","last_clean":"2024-03-24T11:27:13.382571+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:27:13.382571+0000","last_undegraded":"2024-03-24T11:27:13.382571+0000","last_fullsized":"2024-03-24T11:27:13.382571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:25:36.928024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.928024+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.385498+0000","last_change":"2024-03-24T11:26:33.238031+0000","last_active":"2024-03-24T11:27:13.385498+0000","last_peered":"2024-03-24T11:27:13.385498+0000","last_clean":"2024-03-24T11:27:13.385498+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:27:13.385498+0000","last_undegraded":"2024-03-24T11:27:13.385498+0000","last_fullsized":"2024-03-24T11:27:13.385498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.237864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.237864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:40.306806+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.382385+0000","last_change":"2024-03-24T11:25:50.982388+0000","last_active":"2024-03-24T11:27:13.382385+0000","last_peered":"2024-03-24T11:27:13.382385+0000","last_clean":"2024-03-24T11:27:13.382385+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:27:13.382385+0000","last_undegraded":"2024-03-24T11:27:13.382385+0000","last_fullsized":"2024-03-24T11:27:13.382385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:50.982261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:50.982261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.410308+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345323+0000","last_change":"2024-03-24T11:26:33.874445+0000","last_active":"2024-03-24T11:27:13.345323+0000","last_peered":"2024-03-24T11:27:13.345323+0000","last_clean":"2024-03-24T11:27:13.345323+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:27:13.345323+0000","last_undegraded":"2024-03-24T11:27:13.345323+0000","last_fullsized":"2024-03-24T11:27:13.345323+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.874331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.874331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:01.789419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"129'16","reported_seq":302,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346866+0000","last_change":"2024-03-24T11:25:50.138665+0000","last_active":"2024-03-24T11:27:13.346866+0000","last_peered":"2024-03-24T11:27:13.346866+0000","last_clean":"2024-03-24T11:27:13.346866+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:27:13.346866+0000","last_undegraded":"2024-03-24T11:27:13.346866+0000","last_fullsized":"2024-03-24T11:27:13.346866+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'16","last_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:25:50.138485+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340637+0000","last_change":"2024-03-24T11:25:43.229826+0000","last_active":"2024-03-24T11:27:13.340637+0000","last_peered":"2024-03-24T11:27:13.340637+0000","last_clean":"2024-03-24T11:27:13.340637+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:27:13.340637+0000","last_undegraded":"2024-03-24T11:27:13.340637+0000","last_fullsized":"2024-03-24T11:27:13.340637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:43.229470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.229470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":383,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341394+0000","last_change":"2024-03-24T11:25:25.010482+0000","last_active":"2024-03-24T11:27:13.341394+0000","last_peered":"2024-03-24T11:27:13.341394+0000","last_clean":"2024-03-24T11:27:13.341394+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:27:13.341394+0000","last_undegraded":"2024-03-24T11:27:13.341394+0000","last_fullsized":"2024-03-24T11:27:13.341394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:25.010351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:25.010351+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346773+0000","last_change":"2024-03-24T11:25:30.234465+0000","last_active":"2024-03-24T11:27:13.346773+0000","last_peered":"2024-03-24T11:27:13.346773+0000","last_clean":"2024-03-24T11:27:13.346773+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:27:13.346773+0000","last_undegraded":"2024-03-24T11:27:13.346773+0000","last_fullsized":"2024-03-24T11:27:13.346773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:30.234343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:30.234343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":277,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.389575+0000","last_change":"2024-03-24T11:23:48.940813+0000","last_active":"2024-03-24T11:27:13.389575+0000","last_peered":"2024-03-24T11:27:13.389575+0000","last_clean":"2024-03-24T11:27:13.389575+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:27:13.389575+0000","last_undegraded":"2024-03-24T11:27:13.389575+0000","last_fullsized":"2024-03-24T11:27:13.389575+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:23:48.940558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:48.940558+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345572+0000","last_change":"2024-03-24T11:26:39.812393+0000","last_active":"2024-03-24T11:27:13.345572+0000","last_peered":"2024-03-24T11:27:13.345572+0000","last_clean":"2024-03-24T11:27:13.345572+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:27:13.345572+0000","last_undegraded":"2024-03-24T11:27:13.345572+0000","last_fullsized":"2024-03-24T11:27:13.345572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.812259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.728524+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":375,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.389771+0000","last_change":"2024-03-24T11:25:49.322508+0000","last_active":"2024-03-24T11:27:13.389771+0000","last_peered":"2024-03-24T11:27:13.389771+0000","last_clean":"2024-03-24T11:27:13.389771+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:27:13.389771+0000","last_undegraded":"2024-03-24T11:27:13.389771+0000","last_fullsized":"2024-03-24T11:27:13.389771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:49.322383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.322383+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340152+0000","last_change":"2024-03-24T11:26:46.640485+0000","last_active":"2024-03-24T11:27:13.340152+0000","last_peered":"2024-03-24T11:27:13.340152+0000","last_clean":"2024-03-24T11:27:13.340152+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:27:13.340152+0000","last_undegraded":"2024-03-24T11:27:13.340152+0000","last_fullsized":"2024-03-24T11:27:13.340152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.640304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.327035+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":294,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341546+0000","last_change":"2024-03-24T11:25:39.060762+0000","last_active":"2024-03-24T11:27:13.341546+0000","last_peered":"2024-03-24T11:27:13.341546+0000","last_clean":"2024-03-24T11:27:13.341546+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:27:13.341546+0000","last_undegraded":"2024-03-24T11:27:13.341546+0000","last_fullsized":"2024-03-24T11:27:13.341546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:25:39.058804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.058804+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":278,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.378112+0000","last_change":"2024-03-24T11:26:35.783770+0000","last_active":"2024-03-24T11:27:13.378112+0000","last_peered":"2024-03-24T11:27:13.378112+0000","last_clean":"2024-03-24T11:27:13.378112+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:27:13.378112+0000","last_undegraded":"2024-03-24T11:27:13.378112+0000","last_fullsized":"2024-03-24T11:27:13.378112+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:35.783364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:35.783364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.529071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":266,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.387589+0000","last_change":"2024-03-24T11:26:42.302143+0000","last_active":"2024-03-24T11:27:13.387589+0000","last_peered":"2024-03-24T11:27:13.387589+0000","last_clean":"2024-03-24T11:27:13.387589+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:27:13.387589+0000","last_undegraded":"2024-03-24T11:27:13.387589+0000","last_fullsized":"2024-03-24T11:27:13.387589+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.301950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:55.496626+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345625+0000","last_change":"2024-03-24T11:26:27.880943+0000","last_active":"2024-03-24T11:27:13.345625+0000","last_peered":"2024-03-24T11:27:13.345625+0000","last_clean":"2024-03-24T11:27:13.345625+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:27:13.345625+0000","last_undegraded":"2024-03-24T11:27:13.345625+0000","last_fullsized":"2024-03-24T11:27:13.345625+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:27.880817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.880817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:43.623251+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":298,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340772+0000","last_change":"2024-03-24T11:25:38.225135+0000","last_active":"2024-03-24T11:27:13.340772+0000","last_peered":"2024-03-24T11:27:13.340772+0000","last_clean":"2024-03-24T11:27:13.340772+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:27:13.340772+0000","last_undegraded":"2024-03-24T11:27:13.340772+0000","last_fullsized":"2024-03-24T11:27:13.340772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:25:38.225011+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.225011+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:13.392483+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347312+0000","last_change":"2024-03-24T11:25:42.250783+0000","last_active":"2024-03-24T11:27:13.347312+0000","last_peered":"2024-03-24T11:27:13.347312+0000","last_clean":"2024-03-24T11:27:13.347312+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:27:13.347312+0000","last_undegraded":"2024-03-24T11:27:13.347312+0000","last_fullsized":"2024-03-24T11:27:13.347312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:42.250472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:42.250472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938407+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341408+0000","last_change":"2024-03-24T11:25:26.007618+0000","last_active":"2024-03-24T11:27:13.341408+0000","last_peered":"2024-03-24T11:27:13.341408+0000","last_clean":"2024-03-24T11:27:13.341408+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:27:13.341408+0000","last_undegraded":"2024-03-24T11:27:13.341408+0000","last_fullsized":"2024-03-24T11:27:13.341408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:26.007503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:26.007503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":290,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.339201+0000","last_change":"2024-03-24T11:26:30.706204+0000","last_active":"2024-03-24T11:27:13.339201+0000","last_peered":"2024-03-24T11:27:13.339201+0000","last_clean":"2024-03-24T11:27:13.339201+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:27:13.339201+0000","last_undegraded":"2024-03-24T11:27:13.339201+0000","last_fullsized":"2024-03-24T11:27:13.339201+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:30.706071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:30.706071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:50.611182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":271,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.387688+0000","last_change":"2024-03-24T11:23:38.927421+0000","last_active":"2024-03-24T11:27:13.387688+0000","last_peered":"2024-03-24T11:27:13.387688+0000","last_clean":"2024-03-24T11:27:13.387688+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:27:13.387688+0000","last_undegraded":"2024-03-24T11:27:13.387688+0000","last_fullsized":"2024-03-24T11:27:13.387688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-03-24T11:23:38.927200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:23:38.927200+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"184'10420","reported_seq":10499,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.151615+0000","last_change":"2024-03-24T11:25:02.677444+0000","last_active":"2024-03-24T11:27:18.151615+0000","last_peered":"2024-03-24T11:27:18.151615+0000","last_clean":"2024-03-24T11:27:18.151615+0000","last_became_active":"2024-03-24T11:25:02.676873+0000","last_became_peered":"2024-03-24T11:25:02.676873+0000","last_unstale":"2024-03-24T11:27:18.151615+0000","last_undegraded":"2024-03-24T11:27:18.151615+0000","last_fullsized":"2024-03-24T11:27:18.151615+0000","mapping_epoch":150,"log_start":"171'5400","ondisk_log_start":"171'5400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368050176,"num_objects":5616,"num_object_clones":0,"num_object_copies":16848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10420,"num_write_kb":513152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341422+0000","last_change":"2024-03-24T11:25:21.100395+0000","last_active":"2024-03-24T11:27:13.341422+0000","last_peered":"2024-03-24T11:27:13.341422+0000","last_clean":"2024-03-24T11:27:13.341422+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:27:13.341422+0000","last_undegraded":"2024-03-24T11:27:13.341422+0000","last_fullsized":"2024-03-24T11:27:13.341422+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:21.100253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:21.100253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":296,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347176+0000","last_change":"2024-03-24T11:26:34.845593+0000","last_active":"2024-03-24T11:27:13.347176+0000","last_peered":"2024-03-24T11:27:13.347176+0000","last_clean":"2024-03-24T11:27:13.347176+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:27:13.347176+0000","last_undegraded":"2024-03-24T11:27:13.347176+0000","last_fullsized":"2024-03-24T11:27:13.347176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:26:34.845460+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.845460+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.187113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"129'22","reported_seq":310,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340412+0000","last_change":"2024-03-24T11:25:38.096113+0000","last_active":"2024-03-24T11:27:13.340412+0000","last_peered":"2024-03-24T11:27:13.340412+0000","last_clean":"2024-03-24T11:27:13.340412+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:27:13.340412+0000","last_undegraded":"2024-03-24T11:27:13.340412+0000","last_fullsized":"2024-03-24T11:27:13.340412+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'22","last_scrub_stamp":"2024-03-24T11:25:38.096027+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.096027+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":383,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346270+0000","last_change":"2024-03-24T11:26:48.966305+0000","last_active":"2024-03-24T11:27:13.346270+0000","last_peered":"2024-03-24T11:27:13.346270+0000","last_clean":"2024-03-24T11:27:13.346270+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:27:13.346270+0000","last_undegraded":"2024-03-24T11:27:13.346270+0000","last_fullsized":"2024-03-24T11:27:13.346270+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:48.966187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.966187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.360272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"184'10462","reported_seq":10628,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.212898+0000","last_change":"2024-03-24T11:26:46.326986+0000","last_active":"2024-03-24T11:27:18.212898+0000","last_peered":"2024-03-24T11:27:18.212898+0000","last_clean":"2024-03-24T11:27:18.212898+0000","last_became_active":"2024-03-24T11:25:02.675760+0000","last_became_peered":"2024-03-24T11:25:02.675760+0000","last_unstale":"2024-03-24T11:27:18.212898+0000","last_undegraded":"2024-03-24T11:27:18.212898+0000","last_fullsized":"2024-03-24T11:27:18.212898+0000","mapping_epoch":150,"log_start":"169'4600","ondisk_log_start":"169'4600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'8169","last_scrub_stamp":"2024-03-24T11:26:46.326655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.326655+0000","objects_scrubbed":7921,"log_size":5862,"log_dups_size":0,"ondisk_log_size":5862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.124923+0000","scrub_duration":1769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365428736,"num_objects":5576,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10462,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":254,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.386375+0000","last_change":"2024-03-24T11:23:41.875822+0000","last_active":"2024-03-24T11:27:13.386375+0000","last_peered":"2024-03-24T11:27:13.386375+0000","last_clean":"2024-03-24T11:27:13.386375+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:27:13.386375+0000","last_undegraded":"2024-03-24T11:27:13.386375+0000","last_fullsized":"2024-03-24T11:27:13.386375+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:23:41.875525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:23:41.875525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345993+0000","last_change":"2024-03-24T11:25:36.259283+0000","last_active":"2024-03-24T11:27:13.345993+0000","last_peered":"2024-03-24T11:27:13.345993+0000","last_clean":"2024-03-24T11:27:13.345993+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:27:13.345993+0000","last_undegraded":"2024-03-24T11:27:13.345993+0000","last_fullsized":"2024-03-24T11:27:13.345993+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:36.259040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.259040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938573+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":301,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345626+0000","last_change":"2024-03-24T11:26:36.876595+0000","last_active":"2024-03-24T11:27:13.345626+0000","last_peered":"2024-03-24T11:27:13.345626+0000","last_clean":"2024-03-24T11:27:13.345626+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:27:13.345626+0000","last_undegraded":"2024-03-24T11:27:13.345626+0000","last_fullsized":"2024-03-24T11:27:13.345626+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:26:36.876465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.876465+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:37.974940+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.377599+0000","last_change":"2024-03-24T11:26:37.862505+0000","last_active":"2024-03-24T11:27:13.377599+0000","last_peered":"2024-03-24T11:27:13.377599+0000","last_clean":"2024-03-24T11:27:13.377599+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:27:13.377599+0000","last_undegraded":"2024-03-24T11:27:13.377599+0000","last_fullsized":"2024-03-24T11:27:13.377599+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.862389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.862389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.892368+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"184'10621","reported_seq":10762,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.952095+0000","last_change":"2024-03-24T11:27:18.597916+0000","last_active":"2024-03-24T11:27:18.952095+0000","last_peered":"2024-03-24T11:27:18.952095+0000","last_clean":"2024-03-24T11:27:18.952095+0000","last_became_active":"2024-03-24T11:25:02.676144+0000","last_became_peered":"2024-03-24T11:25:02.676144+0000","last_unstale":"2024-03-24T11:27:18.952095+0000","last_undegraded":"2024-03-24T11:27:18.952095+0000","last_fullsized":"2024-03-24T11:27:18.952095+0000","mapping_epoch":150,"log_start":"169'5100","ondisk_log_start":"169'5100","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10592","last_scrub_stamp":"2024-03-24T11:27:18.597618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.597618+0000","objects_scrubbed":5591,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.670378+0000","scrub_duration":3172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356843520,"num_objects":5445,"num_object_clones":0,"num_object_copies":16335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10621,"num_write_kb":514112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346014+0000","last_change":"2024-03-24T11:25:33.186360+0000","last_active":"2024-03-24T11:27:13.346014+0000","last_peered":"2024-03-24T11:27:13.346014+0000","last_clean":"2024-03-24T11:27:13.346014+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:27:13.346014+0000","last_undegraded":"2024-03-24T11:27:13.346014+0000","last_fullsized":"2024-03-24T11:27:13.346014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:33.186253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:33.186253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938600+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.377399+0000","last_change":"2024-03-24T11:26:41.913283+0000","last_active":"2024-03-24T11:27:13.377399+0000","last_peered":"2024-03-24T11:27:13.377399+0000","last_clean":"2024-03-24T11:27:13.377399+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:27:13.377399+0000","last_undegraded":"2024-03-24T11:27:13.377399+0000","last_fullsized":"2024-03-24T11:27:13.377399+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.913165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.913165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.067615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347607+0000","last_change":"2024-03-24T11:26:44.921628+0000","last_active":"2024-03-24T11:27:13.347607+0000","last_peered":"2024-03-24T11:27:13.347607+0000","last_clean":"2024-03-24T11:27:13.347607+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:27:13.347607+0000","last_undegraded":"2024-03-24T11:27:13.347607+0000","last_fullsized":"2024-03-24T11:27:13.347607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:44.921512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:44.921512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:52.518225+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"88'28","reported_seq":326,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.339670+0000","last_change":"2024-03-24T11:26:34.694013+0000","last_active":"2024-03-24T11:27:13.339670+0000","last_peered":"2024-03-24T11:27:13.339670+0000","last_clean":"2024-03-24T11:27:13.339670+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:27:13.339670+0000","last_undegraded":"2024-03-24T11:27:13.339670+0000","last_fullsized":"2024-03-24T11:27:13.339670+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'28","last_scrub_stamp":"2024-03-24T11:26:34.693862+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.693862+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:54.976521+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"184'10404","reported_seq":10569,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.479781+0000","last_change":"2024-03-24T11:26:44.358566+0000","last_active":"2024-03-24T11:27:18.479781+0000","last_peered":"2024-03-24T11:27:18.479781+0000","last_clean":"2024-03-24T11:27:18.479781+0000","last_became_active":"2024-03-24T11:25:02.599484+0000","last_became_peered":"2024-03-24T11:25:02.599484+0000","last_unstale":"2024-03-24T11:27:18.479781+0000","last_undegraded":"2024-03-24T11:27:18.479781+0000","last_fullsized":"2024-03-24T11:27:18.479781+0000","mapping_epoch":150,"log_start":"171'5500","ondisk_log_start":"171'5500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7990","last_scrub_stamp":"2024-03-24T11:26:44.358125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:44.358125+0000","objects_scrubbed":7882,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:08.519779+0000","scrub_duration":3569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363593728,"num_objects":5548,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10404,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.342422+0000","last_change":"2024-03-24T11:26:26.644711+0000","last_active":"2024-03-24T11:27:13.342422+0000","last_peered":"2024-03-24T11:27:13.342422+0000","last_clean":"2024-03-24T11:27:13.342422+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:27:13.342422+0000","last_undegraded":"2024-03-24T11:27:13.342422+0000","last_fullsized":"2024-03-24T11:27:13.342422+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.644597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.644597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.058406+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.378051+0000","last_change":"2024-03-24T11:26:26.849730+0000","last_active":"2024-03-24T11:27:13.378051+0000","last_peered":"2024-03-24T11:27:13.378051+0000","last_clean":"2024-03-24T11:27:13.378051+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:27:13.378051+0000","last_undegraded":"2024-03-24T11:27:13.378051+0000","last_fullsized":"2024-03-24T11:27:13.378051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.849557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.849557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:50.544052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":288,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341195+0000","last_change":"2024-03-24T11:25:33.078676+0000","last_active":"2024-03-24T11:27:13.341195+0000","last_peered":"2024-03-24T11:27:13.341195+0000","last_clean":"2024-03-24T11:27:13.341195+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:27:13.341195+0000","last_undegraded":"2024-03-24T11:27:13.341195+0000","last_fullsized":"2024-03-24T11:27:13.341195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:25:33.078550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:33.078550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":294,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340426+0000","last_change":"2024-03-24T11:26:29.507764+0000","last_active":"2024-03-24T11:27:13.340426+0000","last_peered":"2024-03-24T11:27:13.340426+0000","last_clean":"2024-03-24T11:27:13.340426+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:27:13.340426+0000","last_undegraded":"2024-03-24T11:27:13.340426+0000","last_fullsized":"2024-03-24T11:27:13.340426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:26:29.507604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.507604+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:38.038415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"184'10687","reported_seq":10836,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.145393+0000","last_change":"2024-03-24T11:27:15.950757+0000","last_active":"2024-03-24T11:27:18.145393+0000","last_peered":"2024-03-24T11:27:18.145393+0000","last_clean":"2024-03-24T11:27:18.145393+0000","last_became_active":"2024-03-24T11:25:02.675496+0000","last_became_peered":"2024-03-24T11:25:02.675496+0000","last_unstale":"2024-03-24T11:27:18.145393+0000","last_undegraded":"2024-03-24T11:27:18.145393+0000","last_fullsized":"2024-03-24T11:27:18.145393+0000","mapping_epoch":150,"log_start":"171'5600","ondisk_log_start":"171'5600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10572","last_scrub_stamp":"2024-03-24T11:27:15.950518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:15.950518+0000","objects_scrubbed":5747,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.068187+0000","scrub_duration":1924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364576768,"num_objects":5563,"num_object_clones":0,"num_object_copies":16689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":412,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.343363+0000","last_change":"2024-03-24T11:25:48.067965+0000","last_active":"2024-03-24T11:27:13.343363+0000","last_peered":"2024-03-24T11:27:13.343363+0000","last_clean":"2024-03-24T11:27:13.343363+0000","last_became_active":"2024-03-24T11:24:29.093656+0000","last_became_peered":"2024-03-24T11:24:29.093656+0000","last_unstale":"2024-03-24T11:27:13.343363+0000","last_undegraded":"2024-03-24T11:27:13.343363+0000","last_fullsized":"2024-03-24T11:27:13.343363+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:48.067828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:48.067828+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.387115+0000","last_change":"2024-03-24T11:26:40.231258+0000","last_active":"2024-03-24T11:27:13.387115+0000","last_peered":"2024-03-24T11:27:13.387115+0000","last_clean":"2024-03-24T11:27:13.387115+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:27:13.387115+0000","last_undegraded":"2024-03-24T11:27:13.387115+0000","last_fullsized":"2024-03-24T11:27:13.387115+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:40.231135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:40.231135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:43.325157+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":294,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340919+0000","last_change":"2024-03-24T11:25:35.099894+0000","last_active":"2024-03-24T11:27:13.340919+0000","last_peered":"2024-03-24T11:27:13.340919+0000","last_clean":"2024-03-24T11:27:13.340919+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:27:13.340919+0000","last_undegraded":"2024-03-24T11:27:13.340919+0000","last_fullsized":"2024-03-24T11:27:13.340919+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:25:35.099746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.099746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":283,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346661+0000","last_change":"2024-03-24T11:26:31.873008+0000","last_active":"2024-03-24T11:27:13.346661+0000","last_peered":"2024-03-24T11:27:13.346661+0000","last_clean":"2024-03-24T11:27:13.346661+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:27:13.346661+0000","last_undegraded":"2024-03-24T11:27:13.346661+0000","last_fullsized":"2024-03-24T11:27:13.346661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:31.872524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:31.872524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:40.048777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":436,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.958327+0000","last_change":"2024-03-24T11:27:13.958327+0000","last_active":"2024-03-24T11:27:13.958327+0000","last_peered":"2024-03-24T11:27:13.958327+0000","last_clean":"2024-03-24T11:27:13.958327+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:27:13.958327+0000","last_undegraded":"2024-03-24T11:27:13.958327+0000","last_fullsized":"2024-03-24T11:27:13.958327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:27:13.958181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:13.958181+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.681771+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":275,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340303+0000","last_change":"2024-03-24T11:26:38.524661+0000","last_active":"2024-03-24T11:27:13.340303+0000","last_peered":"2024-03-24T11:27:13.340303+0000","last_clean":"2024-03-24T11:27:13.340303+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:27:13.340303+0000","last_undegraded":"2024-03-24T11:27:13.340303+0000","last_fullsized":"2024-03-24T11:27:13.340303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:38.524541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:38.524541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.968194+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"184'10788","reported_seq":10866,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.663709+0000","last_change":"2024-03-24T11:25:02.601118+0000","last_active":"2024-03-24T11:27:18.663709+0000","last_peered":"2024-03-24T11:27:18.663709+0000","last_clean":"2024-03-24T11:27:18.663709+0000","last_became_active":"2024-03-24T11:25:02.600777+0000","last_became_peered":"2024-03-24T11:25:02.600777+0000","last_unstale":"2024-03-24T11:27:18.663709+0000","last_undegraded":"2024-03-24T11:27:18.663709+0000","last_fullsized":"2024-03-24T11:27:18.663709+0000","mapping_epoch":150,"log_start":"173'6300","ondisk_log_start":"173'6300","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371326976,"num_objects":5666,"num_object_clones":0,"num_object_copies":16998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10788,"num_write_kb":526528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":552,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345924+0000","last_change":"2024-03-24T11:25:49.282133+0000","last_active":"2024-03-24T11:27:13.345924+0000","last_peered":"2024-03-24T11:27:13.345924+0000","last_clean":"2024-03-24T11:27:13.345924+0000","last_became_active":"2024-03-24T11:24:29.092038+0000","last_became_peered":"2024-03-24T11:24:29.092038+0000","last_unstale":"2024-03-24T11:27:13.345924+0000","last_undegraded":"2024-03-24T11:27:13.345924+0000","last_fullsized":"2024-03-24T11:27:13.345924+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:49.282048+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:25:49.282048+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.282048+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347692+0000","last_change":"2024-03-24T11:25:44.288277+0000","last_active":"2024-03-24T11:27:13.347692+0000","last_peered":"2024-03-24T11:27:13.347692+0000","last_clean":"2024-03-24T11:27:13.347692+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:27:13.347692+0000","last_undegraded":"2024-03-24T11:27:13.347692+0000","last_fullsized":"2024-03-24T11:27:13.347692+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:25:44.288152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938547+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":291,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.382033+0000","last_change":"2024-03-24T11:26:25.702207+0000","last_active":"2024-03-24T11:27:13.382033+0000","last_peered":"2024-03-24T11:27:13.382033+0000","last_clean":"2024-03-24T11:27:13.382033+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:27:13.382033+0000","last_undegraded":"2024-03-24T11:27:13.382033+0000","last_fullsized":"2024-03-24T11:27:13.382033+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:25.702101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:25.702101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:32.898822+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":292,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.381649+0000","last_change":"2024-03-24T11:25:36.710275+0000","last_active":"2024-03-24T11:27:13.381649+0000","last_peered":"2024-03-24T11:27:13.381649+0000","last_clean":"2024-03-24T11:27:13.381649+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:27:13.381649+0000","last_undegraded":"2024-03-24T11:27:13.381649+0000","last_fullsized":"2024-03-24T11:27:13.381649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:36.710123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.710123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.984029+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":280,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.378738+0000","last_change":"2024-03-24T11:25:45.765628+0000","last_active":"2024-03-24T11:27:13.378738+0000","last_peered":"2024-03-24T11:27:13.378738+0000","last_clean":"2024-03-24T11:27:13.378738+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:27:13.378738+0000","last_undegraded":"2024-03-24T11:27:13.378738+0000","last_fullsized":"2024-03-24T11:27:13.378738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:25:45.764917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:45.764917+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.984095+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.377937+0000","last_change":"2024-03-24T11:26:29.820470+0000","last_active":"2024-03-24T11:27:13.377937+0000","last_peered":"2024-03-24T11:27:13.377937+0000","last_clean":"2024-03-24T11:27:13.377937+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:27:13.377937+0000","last_undegraded":"2024-03-24T11:27:13.377937+0000","last_fullsized":"2024-03-24T11:27:13.377937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:29.820240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.820240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:41.262557+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"184'10303","reported_seq":10468,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.100170+0000","last_change":"2024-03-24T11:26:48.313995+0000","last_active":"2024-03-24T11:27:18.100170+0000","last_peered":"2024-03-24T11:27:18.100170+0000","last_clean":"2024-03-24T11:27:18.100170+0000","last_became_active":"2024-03-24T11:25:02.600137+0000","last_became_peered":"2024-03-24T11:25:02.600137+0000","last_unstale":"2024-03-24T11:27:18.100170+0000","last_undegraded":"2024-03-24T11:27:18.100170+0000","last_fullsized":"2024-03-24T11:27:18.100170+0000","mapping_epoch":150,"log_start":"169'4500","ondisk_log_start":"169'4500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8126","last_scrub_stamp":"2024-03-24T11:26:48.313801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.313801+0000","objects_scrubbed":7677,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.465833+0000","scrub_duration":2728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354877440,"num_objects":5415,"num_object_clones":0,"num_object_copies":16245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":278,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341354+0000","last_change":"2024-03-24T11:26:49.074595+0000","last_active":"2024-03-24T11:27:13.341354+0000","last_peered":"2024-03-24T11:27:13.341354+0000","last_clean":"2024-03-24T11:27:13.341354+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:27:13.341354+0000","last_undegraded":"2024-03-24T11:27:13.341354+0000","last_fullsized":"2024-03-24T11:27:13.341354+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.074485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:10.184136+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":545,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346196+0000","last_change":"2024-03-24T11:26:36.872094+0000","last_active":"2024-03-24T11:27:13.346196+0000","last_peered":"2024-03-24T11:27:13.346196+0000","last_clean":"2024-03-24T11:27:13.346196+0000","last_became_active":"2024-03-24T11:24:23.650976+0000","last_became_peered":"2024-03-24T11:24:23.650976+0000","last_unstale":"2024-03-24T11:27:13.346196+0000","last_undegraded":"2024-03-24T11:27:13.346196+0000","last_fullsized":"2024-03-24T11:27:13.346196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:36.871960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.871960+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:41.364084+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":281,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.385959+0000","last_change":"2024-03-24T11:27:12.329869+0000","last_active":"2024-03-24T11:27:13.385959+0000","last_peered":"2024-03-24T11:27:13.385959+0000","last_clean":"2024-03-24T11:27:13.385959+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:27:13.385959+0000","last_undegraded":"2024-03-24T11:27:13.385959+0000","last_fullsized":"2024-03-24T11:27:13.385959+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:35.227538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:35.227538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:35.657337+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"176'24","reported_seq":418,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.825961+0000","last_change":"2024-03-24T11:26:37.676875+0000","last_active":"2024-03-24T11:27:13.825961+0000","last_peered":"2024-03-24T11:27:13.825961+0000","last_clean":"2024-03-24T11:27:13.825961+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:27:13.825961+0000","last_undegraded":"2024-03-24T11:27:13.825961+0000","last_fullsized":"2024-03-24T11:27:13.825961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'24","last_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.676764+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:48.350269+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346936+0000","last_change":"2024-03-24T11:27:11.941555+0000","last_active":"2024-03-24T11:27:13.346936+0000","last_peered":"2024-03-24T11:27:13.346936+0000","last_clean":"2024-03-24T11:27:13.346936+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:27:13.346936+0000","last_undegraded":"2024-03-24T11:27:13.346936+0000","last_fullsized":"2024-03-24T11:27:13.346936+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:11.941414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.941414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.864708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"184'10310","reported_seq":10475,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.425761+0000","last_change":"2024-03-24T11:26:42.718381+0000","last_active":"2024-03-24T11:27:18.425761+0000","last_peered":"2024-03-24T11:27:18.425761+0000","last_clean":"2024-03-24T11:27:18.425761+0000","last_became_active":"2024-03-24T11:25:02.674959+0000","last_became_peered":"2024-03-24T11:25:02.674959+0000","last_unstale":"2024-03-24T11:27:18.425761+0000","last_undegraded":"2024-03-24T11:27:18.425761+0000","last_fullsized":"2024-03-24T11:27:18.425761+0000","mapping_epoch":150,"log_start":"171'5400","ondisk_log_start":"171'5400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7858","last_scrub_stamp":"2024-03-24T11:26:42.718210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.718210+0000","objects_scrubbed":7774,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.515083+0000","scrub_duration":2944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363986944,"num_objects":5554,"num_object_clones":0,"num_object_copies":16662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10310,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":412,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.393099+0000","last_change":"2024-03-24T11:26:27.707698+0000","last_active":"2024-03-24T11:27:13.393099+0000","last_peered":"2024-03-24T11:27:13.393099+0000","last_clean":"2024-03-24T11:27:13.393099+0000","last_became_active":"2024-03-24T11:24:29.087723+0000","last_became_peered":"2024-03-24T11:24:29.087723+0000","last_unstale":"2024-03-24T11:27:13.393099+0000","last_undegraded":"2024-03-24T11:27:13.393099+0000","last_fullsized":"2024-03-24T11:27:13.393099+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:27.707575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.707575+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:30.342451+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":278,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.382187+0000","last_change":"2024-03-24T11:26:42.779061+0000","last_active":"2024-03-24T11:27:13.382187+0000","last_peered":"2024-03-24T11:27:13.382187+0000","last_clean":"2024-03-24T11:27:13.382187+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:27:13.382187+0000","last_undegraded":"2024-03-24T11:27:13.382187+0000","last_fullsized":"2024-03-24T11:27:13.382187+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.778835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.166597+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":286,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.342371+0000","last_change":"2024-03-24T11:26:41.522630+0000","last_active":"2024-03-24T11:27:13.342371+0000","last_peered":"2024-03-24T11:27:13.342371+0000","last_clean":"2024-03-24T11:27:13.342371+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:27:13.342371+0000","last_undegraded":"2024-03-24T11:27:13.342371+0000","last_fullsized":"2024-03-24T11:27:13.342371+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.522032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:42.372094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":784,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:16.148078+0000","last_change":"2024-03-24T11:25:27.279271+0000","last_active":"2024-03-24T11:27:16.148078+0000","last_peered":"2024-03-24T11:27:16.148078+0000","last_clean":"2024-03-24T11:27:16.148078+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:27:16.148078+0000","last_undegraded":"2024-03-24T11:27:16.148078+0000","last_fullsized":"2024-03-24T11:27:16.148078+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:25:27.279145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:25:27.279145+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":497,"num_read_kb":492,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":296,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340655+0000","last_change":"2024-03-24T11:27:11.451262+0000","last_active":"2024-03-24T11:27:13.340655+0000","last_peered":"2024-03-24T11:27:13.340655+0000","last_clean":"2024-03-24T11:27:13.340655+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:27:13.340655+0000","last_undegraded":"2024-03-24T11:27:13.340655+0000","last_fullsized":"2024-03-24T11:27:13.340655+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:27:11.451128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.451128+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.371691+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:17.037117+0000","last_change":"2024-03-24T11:27:17.037155+0000","last_active":"2024-03-24T11:27:17.037117+0000","last_peered":"2024-03-24T11:27:17.037117+0000","last_clean":"2024-03-24T11:27:17.037117+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:27:17.037117+0000","last_undegraded":"2024-03-24T11:27:17.037117+0000","last_fullsized":"2024-03-24T11:27:17.037117+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.037043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.037043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.898329+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"184'10483","reported_seq":10562,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.670388+0000","last_change":"2024-03-24T11:25:02.677146+0000","last_active":"2024-03-24T11:27:18.670388+0000","last_peered":"2024-03-24T11:27:18.670388+0000","last_clean":"2024-03-24T11:27:18.670388+0000","last_became_active":"2024-03-24T11:25:02.676840+0000","last_became_peered":"2024-03-24T11:25:02.676840+0000","last_unstale":"2024-03-24T11:27:18.670388+0000","last_undegraded":"2024-03-24T11:27:18.670388+0000","last_fullsized":"2024-03-24T11:27:18.670388+0000","mapping_epoch":150,"log_start":"173'6000","ondisk_log_start":"173'6000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368902144,"num_objects":5629,"num_object_clones":0,"num_object_copies":16887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10483,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":418,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340755+0000","last_change":"2024-03-24T11:26:43.595952+0000","last_active":"2024-03-24T11:27:13.340755+0000","last_peered":"2024-03-24T11:27:13.340755+0000","last_clean":"2024-03-24T11:27:13.340755+0000","last_became_active":"2024-03-24T11:24:29.081667+0000","last_became_peered":"2024-03-24T11:24:29.081667+0000","last_unstale":"2024-03-24T11:27:13.340755+0000","last_undegraded":"2024-03-24T11:27:13.340755+0000","last_fullsized":"2024-03-24T11:27:13.340755+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:43.595832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.595832+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.040167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.388209+0000","last_change":"2024-03-24T11:25:49.059451+0000","last_active":"2024-03-24T11:27:13.388209+0000","last_peered":"2024-03-24T11:27:13.388209+0000","last_clean":"2024-03-24T11:27:13.388209+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:27:13.388209+0000","last_undegraded":"2024-03-24T11:27:13.388209+0000","last_fullsized":"2024-03-24T11:27:13.388209+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:49.058968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.058968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":291,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.383771+0000","last_change":"2024-03-24T11:25:50.022235+0000","last_active":"2024-03-24T11:27:13.383771+0000","last_peered":"2024-03-24T11:27:13.383771+0000","last_clean":"2024-03-24T11:27:13.383771+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:27:13.383771+0000","last_undegraded":"2024-03-24T11:27:13.383771+0000","last_fullsized":"2024-03-24T11:27:13.383771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:50.022117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:50.022117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":307,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340299+0000","last_change":"2024-03-24T11:25:34.223822+0000","last_active":"2024-03-24T11:27:13.340299+0000","last_peered":"2024-03-24T11:27:13.340299+0000","last_clean":"2024-03-24T11:27:13.340299+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:27:13.340299+0000","last_undegraded":"2024-03-24T11:27:13.340299+0000","last_fullsized":"2024-03-24T11:27:13.340299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-24T11:25:34.223668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:34.223668+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":283,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.339820+0000","last_change":"2024-03-24T11:26:33.660212+0000","last_active":"2024-03-24T11:27:13.339820+0000","last_peered":"2024-03-24T11:27:13.339820+0000","last_clean":"2024-03-24T11:27:13.339820+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:27:13.339820+0000","last_undegraded":"2024-03-24T11:27:13.339820+0000","last_fullsized":"2024-03-24T11:27:13.339820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.660136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:05.186078+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.660136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:34.463306+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"184'10503","reported_seq":10607,"reported_epoch":184,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T11:27:18.959388+0000","last_change":"2024-03-24T11:27:17.395949+0000","last_active":"2024-03-24T11:27:18.959388+0000","last_peered":"2024-03-24T11:27:18.959388+0000","last_clean":"2024-03-24T11:27:18.959388+0000","last_became_active":"2024-03-24T11:25:02.676938+0000","last_became_peered":"2024-03-24T11:25:02.676938+0000","last_unstale":"2024-03-24T11:27:18.959388+0000","last_undegraded":"2024-03-24T11:27:18.959388+0000","last_fullsized":"2024-03-24T11:27:18.959388+0000","mapping_epoch":150,"log_start":"169'5000","ondisk_log_start":"169'5000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":2349,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360775680,"num_objects":5505,"num_object_clones":0,"num_object_copies":16515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":512256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":418,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340932+0000","last_change":"2024-03-24T11:26:32.673765+0000","last_active":"2024-03-24T11:27:13.340932+0000","last_peered":"2024-03-24T11:27:13.340932+0000","last_clean":"2024-03-24T11:27:13.340932+0000","last_became_active":"2024-03-24T11:24:29.117106+0000","last_became_peered":"2024-03-24T11:24:29.117106+0000","last_unstale":"2024-03-24T11:27:13.340932+0000","last_undegraded":"2024-03-24T11:27:13.340932+0000","last_fullsized":"2024-03-24T11:27:13.340932+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:32.673525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.673525+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:53.552032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":264,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.339922+0000","last_change":"2024-03-24T11:25:28.354779+0000","last_active":"2024-03-24T11:27:13.339922+0000","last_peered":"2024-03-24T11:27:13.339922+0000","last_clean":"2024-03-24T11:27:13.339922+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:27:13.339922+0000","last_undegraded":"2024-03-24T11:27:13.339922+0000","last_fullsized":"2024-03-24T11:27:13.339922+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:25:28.354644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:28.354644+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":286,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341489+0000","last_change":"2024-03-24T11:25:19.123540+0000","last_active":"2024-03-24T11:27:13.341489+0000","last_peered":"2024-03-24T11:27:13.341489+0000","last_clean":"2024-03-24T11:27:13.341489+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:27:13.341489+0000","last_undegraded":"2024-03-24T11:27:13.341489+0000","last_fullsized":"2024-03-24T11:27:13.341489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:19.123468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:19.123468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":302,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.382514+0000","last_change":"2024-03-24T11:26:49.790900+0000","last_active":"2024-03-24T11:27:13.382514+0000","last_peered":"2024-03-24T11:27:13.382514+0000","last_clean":"2024-03-24T11:27:13.382514+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:27:13.382514+0000","last_undegraded":"2024-03-24T11:27:13.382514+0000","last_fullsized":"2024-03-24T11:27:13.382514+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:26:49.790753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.790753+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.066698+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":385,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.826075+0000","last_change":"2024-03-24T11:25:40.204011+0000","last_active":"2024-03-24T11:27:18.826075+0000","last_peered":"2024-03-24T11:27:18.826075+0000","last_clean":"2024-03-24T11:27:18.826075+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:27:18.826075+0000","last_undegraded":"2024-03-24T11:27:18.826075+0000","last_fullsized":"2024-03-24T11:27:18.826075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:40.203874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.203874+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"184'10402","reported_seq":10567,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.168293+0000","last_change":"2024-03-24T11:26:49.441112+0000","last_active":"2024-03-24T11:27:18.168293+0000","last_peered":"2024-03-24T11:27:18.168293+0000","last_clean":"2024-03-24T11:27:18.168293+0000","last_became_active":"2024-03-24T11:25:02.675745+0000","last_became_peered":"2024-03-24T11:25:02.675745+0000","last_unstale":"2024-03-24T11:27:18.168293+0000","last_undegraded":"2024-03-24T11:27:18.168293+0000","last_fullsized":"2024-03-24T11:27:18.168293+0000","mapping_epoch":150,"log_start":"171'5400","ondisk_log_start":"171'5400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8346","last_scrub_stamp":"2024-03-24T11:26:49.440843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.440843+0000","objects_scrubbed":7686,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.854214+0000","scrub_duration":2353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363135000,"num_objects":5542,"num_object_clones":0,"num_object_copies":16626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10402,"num_write_kb":510145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":404,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.388127+0000","last_change":"2024-03-24T11:24:30.117954+0000","last_active":"2024-03-24T11:27:13.388127+0000","last_peered":"2024-03-24T11:27:13.388127+0000","last_clean":"2024-03-24T11:27:13.388127+0000","last_became_active":"2024-03-24T11:24:29.087468+0000","last_became_peered":"2024-03-24T11:24:29.087468+0000","last_unstale":"2024-03-24T11:27:13.388127+0000","last_undegraded":"2024-03-24T11:27:13.388127+0000","last_fullsized":"2024-03-24T11:27:13.388127+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:24:30.117825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:24:30.117825+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346598+0000","last_change":"2024-03-24T11:25:26.285547+0000","last_active":"2024-03-24T11:27:13.346598+0000","last_peered":"2024-03-24T11:27:13.346598+0000","last_clean":"2024-03-24T11:27:13.346598+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:27:13.346598+0000","last_undegraded":"2024-03-24T11:27:13.346598+0000","last_fullsized":"2024-03-24T11:27:13.346598+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:26.285465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:26.285465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:14.938379+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":283,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340718+0000","last_change":"2024-03-24T11:25:44.174500+0000","last_active":"2024-03-24T11:27:13.340718+0000","last_peered":"2024-03-24T11:27:13.340718+0000","last_clean":"2024-03-24T11:27:13.340718+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:27:13.340718+0000","last_undegraded":"2024-03-24T11:27:13.340718+0000","last_fullsized":"2024-03-24T11:27:13.340718+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:25:44.174271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:44.174271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:13.392519+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":300,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:15.813035+0000","last_change":"2024-03-24T11:27:15.813035+0000","last_active":"2024-03-24T11:27:15.813035+0000","last_peered":"2024-03-24T11:27:15.813035+0000","last_clean":"2024-03-24T11:27:15.813035+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:27:15.813035+0000","last_undegraded":"2024-03-24T11:27:15.813035+0000","last_fullsized":"2024-03-24T11:27:15.813035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-03-24T11:27:15.812897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:15.812897+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:19.144490+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":368,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.390282+0000","last_change":"2024-03-24T11:25:20.232658+0000","last_active":"2024-03-24T11:27:13.390282+0000","last_peered":"2024-03-24T11:27:13.390282+0000","last_clean":"2024-03-24T11:27:13.390282+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:27:13.390282+0000","last_undegraded":"2024-03-24T11:27:13.390282+0000","last_fullsized":"2024-03-24T11:27:13.390282+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:20.232540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:20.232540+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"184'10249","reported_seq":10393,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.944165+0000","last_change":"2024-03-24T11:27:18.681143+0000","last_active":"2024-03-24T11:27:18.944165+0000","last_peered":"2024-03-24T11:27:18.944165+0000","last_clean":"2024-03-24T11:27:18.944165+0000","last_became_active":"2024-03-24T11:25:02.675706+0000","last_became_peered":"2024-03-24T11:25:02.675706+0000","last_unstale":"2024-03-24T11:27:18.944165+0000","last_undegraded":"2024-03-24T11:27:18.944165+0000","last_fullsized":"2024-03-24T11:27:18.944165+0000","mapping_epoch":150,"log_start":"169'4700","ondisk_log_start":"169'4700","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10233","last_scrub_stamp":"2024-03-24T11:27:18.680945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.680945+0000","objects_scrubbed":5603,"log_size":5549,"log_dups_size":0,"ondisk_log_size":5549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:20.438526+0000","scrub_duration":2239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362348544,"num_objects":5529,"num_object_clones":0,"num_object_copies":16587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10249,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":412,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346614+0000","last_change":"2024-03-24T11:26:29.924513+0000","last_active":"2024-03-24T11:27:13.346614+0000","last_peered":"2024-03-24T11:27:13.346614+0000","last_clean":"2024-03-24T11:27:13.346614+0000","last_became_active":"2024-03-24T11:24:29.082786+0000","last_became_peered":"2024-03-24T11:24:29.082786+0000","last_unstale":"2024-03-24T11:27:13.346614+0000","last_undegraded":"2024-03-24T11:27:13.346614+0000","last_fullsized":"2024-03-24T11:27:13.346614+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:29.924076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.924076+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:39.946530+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.346114+0000","last_change":"2024-03-24T11:25:42.060954+0000","last_active":"2024-03-24T11:27:13.346114+0000","last_peered":"2024-03-24T11:27:13.346114+0000","last_clean":"2024-03-24T11:27:13.346114+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:27:13.346114+0000","last_undegraded":"2024-03-24T11:27:13.346114+0000","last_fullsized":"2024-03-24T11:27:13.346114+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:42.060840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:49.118465+0000","last_clean_scrub_stamp":"2024-03-24T11:25:42.060840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.387387+0000","last_change":"2024-03-24T11:26:39.266713+0000","last_active":"2024-03-24T11:27:13.387387+0000","last_peered":"2024-03-24T11:27:13.387387+0000","last_clean":"2024-03-24T11:27:13.387387+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:27:13.387387+0000","last_undegraded":"2024-03-24T11:27:13.387387+0000","last_fullsized":"2024-03-24T11:27:13.387387+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.265692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.265692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.928492+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":295,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.390151+0000","last_change":"2024-03-24T11:25:35.339628+0000","last_active":"2024-03-24T11:27:13.390151+0000","last_peered":"2024-03-24T11:27:13.390151+0000","last_clean":"2024-03-24T11:27:13.390151+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:27:13.390151+0000","last_undegraded":"2024-03-24T11:27:13.390151+0000","last_fullsized":"2024-03-24T11:27:13.390151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-24T11:25:35.339383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.339383+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":275,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345245+0000","last_change":"2024-03-24T11:26:28.950793+0000","last_active":"2024-03-24T11:27:13.345245+0000","last_peered":"2024-03-24T11:27:13.345245+0000","last_clean":"2024-03-24T11:27:13.345245+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:27:13.345245+0000","last_undegraded":"2024-03-24T11:27:13.345245+0000","last_fullsized":"2024-03-24T11:27:13.345245+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.950670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.950670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:30.361274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"184'10477","reported_seq":10637,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.477876+0000","last_change":"2024-03-24T11:26:34.774580+0000","last_active":"2024-03-24T11:27:18.477876+0000","last_peered":"2024-03-24T11:27:18.477876+0000","last_clean":"2024-03-24T11:27:18.477876+0000","last_became_active":"2024-03-24T11:25:02.600315+0000","last_became_peered":"2024-03-24T11:25:02.600315+0000","last_unstale":"2024-03-24T11:27:18.477876+0000","last_undegraded":"2024-03-24T11:27:18.477876+0000","last_fullsized":"2024-03-24T11:27:18.477876+0000","mapping_epoch":150,"log_start":"171'5600","ondisk_log_start":"171'5600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7439","last_scrub_stamp":"2024-03-24T11:26:34.774083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.774083+0000","objects_scrubbed":7347,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:52.198850+0000","scrub_duration":2963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367198208,"num_objects":5603,"num_object_clones":0,"num_object_copies":16809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":412,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340604+0000","last_change":"2024-03-24T11:25:40.019964+0000","last_active":"2024-03-24T11:27:13.340604+0000","last_peered":"2024-03-24T11:27:13.340604+0000","last_clean":"2024-03-24T11:27:13.340604+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:27:13.340604+0000","last_undegraded":"2024-03-24T11:27:13.340604+0000","last_fullsized":"2024-03-24T11:27:13.340604+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:40.019881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.019881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341131+0000","last_change":"2024-03-24T11:25:20.173363+0000","last_active":"2024-03-24T11:27:13.341131+0000","last_peered":"2024-03-24T11:27:13.341131+0000","last_clean":"2024-03-24T11:27:13.341131+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:27:13.341131+0000","last_undegraded":"2024-03-24T11:27:13.341131+0000","last_fullsized":"2024-03-24T11:27:13.341131+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:20.173220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:20.173220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":383,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347494+0000","last_change":"2024-03-24T11:26:42.827014+0000","last_active":"2024-03-24T11:27:13.347494+0000","last_peered":"2024-03-24T11:27:13.347494+0000","last_clean":"2024-03-24T11:27:13.347494+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:27:13.347494+0000","last_undegraded":"2024-03-24T11:27:13.347494+0000","last_fullsized":"2024-03-24T11:27:13.347494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:42.826873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.826873+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:04.232390+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.390089+0000","last_change":"2024-03-24T11:25:22.291763+0000","last_active":"2024-03-24T11:27:13.390089+0000","last_peered":"2024-03-24T11:27:13.390089+0000","last_clean":"2024-03-24T11:27:13.390089+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:27:13.390089+0000","last_undegraded":"2024-03-24T11:27:13.390089+0000","last_fullsized":"2024-03-24T11:27:13.390089+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:22.291490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:25:22.291490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"89'22","reported_seq":318,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.390002+0000","last_change":"2024-03-24T11:26:46.757168+0000","last_active":"2024-03-24T11:27:13.390002+0000","last_peered":"2024-03-24T11:27:13.390002+0000","last_clean":"2024-03-24T11:27:13.390002+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:27:13.390002+0000","last_undegraded":"2024-03-24T11:27:13.390002+0000","last_fullsized":"2024-03-24T11:27:13.390002+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-03-24T11:26:46.757070+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.757070+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.851634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"184'10229","reported_seq":10393,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.143370+0000","last_change":"2024-03-24T11:26:48.243638+0000","last_active":"2024-03-24T11:27:18.143370+0000","last_peered":"2024-03-24T11:27:18.143370+0000","last_clean":"2024-03-24T11:27:18.143370+0000","last_became_active":"2024-03-24T11:25:02.599329+0000","last_became_peered":"2024-03-24T11:25:02.599329+0000","last_unstale":"2024-03-24T11:27:18.143370+0000","last_undegraded":"2024-03-24T11:27:18.143370+0000","last_fullsized":"2024-03-24T11:27:18.143370+0000","mapping_epoch":150,"log_start":"172'5600","ondisk_log_start":"172'5600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8117","last_scrub_stamp":"2024-03-24T11:26:48.243210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.243210+0000","objects_scrubbed":7656,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.207893+0000","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359858176,"num_objects":5491,"num_object_clones":0,"num_object_copies":16473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":262,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:17.423008+0000","last_change":"2024-03-24T11:27:17.423048+0000","last_active":"2024-03-24T11:27:17.423008+0000","last_peered":"2024-03-24T11:27:17.423008+0000","last_clean":"2024-03-24T11:27:17.423008+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:27:17.423008+0000","last_undegraded":"2024-03-24T11:27:17.423008+0000","last_fullsized":"2024-03-24T11:27:17.423008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.422927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.062528+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.341636+0000","last_change":"2024-03-24T11:26:28.105930+0000","last_active":"2024-03-24T11:27:13.341636+0000","last_peered":"2024-03-24T11:27:13.341636+0000","last_clean":"2024-03-24T11:27:13.341636+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:27:13.341636+0000","last_undegraded":"2024-03-24T11:27:13.341636+0000","last_fullsized":"2024-03-24T11:27:13.341636+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.105784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:24:43.042611+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.105784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:48.780495+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345295+0000","last_change":"2024-03-24T11:26:49.883140+0000","last_active":"2024-03-24T11:27:13.345295+0000","last_peered":"2024-03-24T11:27:13.345295+0000","last_clean":"2024-03-24T11:27:13.345295+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:27:13.345295+0000","last_undegraded":"2024-03-24T11:27:13.345295+0000","last_fullsized":"2024-03-24T11:27:13.345295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.883061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.883061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:52.393144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":297,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347346+0000","last_change":"2024-03-24T11:25:43.030583+0000","last_active":"2024-03-24T11:27:13.347346+0000","last_peered":"2024-03-24T11:27:13.347346+0000","last_clean":"2024-03-24T11:27:13.347346+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:27:13.347346+0000","last_undegraded":"2024-03-24T11:27:13.347346+0000","last_fullsized":"2024-03-24T11:27:13.347346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:25:43.030417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.030417+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"184'10453","reported_seq":10616,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.457072+0000","last_change":"2024-03-24T11:26:36.863675+0000","last_active":"2024-03-24T11:27:18.457072+0000","last_peered":"2024-03-24T11:27:18.457072+0000","last_clean":"2024-03-24T11:27:18.457072+0000","last_became_active":"2024-03-24T11:25:02.674932+0000","last_became_peered":"2024-03-24T11:25:02.674932+0000","last_unstale":"2024-03-24T11:27:18.457072+0000","last_undegraded":"2024-03-24T11:27:18.457072+0000","last_fullsized":"2024-03-24T11:27:18.457072+0000","mapping_epoch":150,"log_start":"171'5600","ondisk_log_start":"171'5600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7579","last_scrub_stamp":"2024-03-24T11:26:36.863361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.863361+0000","objects_scrubbed":7488,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:06.565935+0000","scrub_duration":3086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366280704,"num_objects":5589,"num_object_clones":0,"num_object_copies":16767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10453,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":413,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.340232+0000","last_change":"2024-03-24T11:25:43.004840+0000","last_active":"2024-03-24T11:27:13.340232+0000","last_peered":"2024-03-24T11:27:13.340232+0000","last_clean":"2024-03-24T11:27:13.340232+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:27:13.340232+0000","last_undegraded":"2024-03-24T11:27:13.340232+0000","last_fullsized":"2024-03-24T11:27:13.340232+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:25:43.004596+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:25:43.004596+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.004596+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.384130+0000","last_change":"2024-03-24T11:25:25.841724+0000","last_active":"2024-03-24T11:27:13.384130+0000","last_peered":"2024-03-24T11:27:13.384130+0000","last_clean":"2024-03-24T11:27:13.384130+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:27:13.384130+0000","last_undegraded":"2024-03-24T11:27:13.384130+0000","last_fullsized":"2024-03-24T11:27:13.384130+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:25.841476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:25.841476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.983962+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":282,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.382595+0000","last_change":"2024-03-24T11:26:48.793731+0000","last_active":"2024-03-24T11:27:13.382595+0000","last_peered":"2024-03-24T11:27:13.382595+0000","last_clean":"2024-03-24T11:27:13.382595+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:27:13.382595+0000","last_undegraded":"2024-03-24T11:27:13.382595+0000","last_fullsized":"2024-03-24T11:27:13.382595+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.793614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.316226+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":296,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.339499+0000","last_change":"2024-03-24T11:25:43.745091+0000","last_active":"2024-03-24T11:27:13.339499+0000","last_peered":"2024-03-24T11:27:13.339499+0000","last_clean":"2024-03-24T11:27:13.339499+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:27:13.339499+0000","last_undegraded":"2024-03-24T11:27:13.339499+0000","last_fullsized":"2024-03-24T11:27:13.339499+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:25:43.744958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.744958+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.492046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"88'25","reported_seq":338,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347344+0000","last_change":"2024-03-24T11:25:49.109903+0000","last_active":"2024-03-24T11:27:13.347344+0000","last_peered":"2024-03-24T11:27:13.347344+0000","last_clean":"2024-03-24T11:27:13.347344+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:27:13.347344+0000","last_undegraded":"2024-03-24T11:27:13.347344+0000","last_fullsized":"2024-03-24T11:27:13.347344+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'25","last_scrub_stamp":"2024-03-24T11:25:49.109688+0000","last_deep_scrub":"88'25","last_deep_scrub_stamp":"2024-03-24T11:25:49.109688+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.109688+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:16.118882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"184'10513","reported_seq":10592,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:18.663076+0000","last_change":"2024-03-24T11:25:02.599491+0000","last_active":"2024-03-24T11:27:18.663076+0000","last_peered":"2024-03-24T11:27:18.663076+0000","last_clean":"2024-03-24T11:27:18.663076+0000","last_became_active":"2024-03-24T11:25:02.599177+0000","last_became_peered":"2024-03-24T11:25:02.599177+0000","last_unstale":"2024-03-24T11:27:18.663076+0000","last_undegraded":"2024-03-24T11:27:18.663076+0000","last_fullsized":"2024-03-24T11:27:18.663076+0000","mapping_epoch":150,"log_start":"173'6000","ondisk_log_start":"173'6000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360775680,"num_objects":5505,"num_object_clones":0,"num_object_copies":16515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":512576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":560,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345387+0000","last_change":"2024-03-24T11:27:12.962801+0000","last_active":"2024-03-24T11:27:13.345387+0000","last_peered":"2024-03-24T11:27:13.345387+0000","last_clean":"2024-03-24T11:27:13.345387+0000","last_became_active":"2024-03-24T11:24:29.091858+0000","last_became_peered":"2024-03-24T11:24:29.091858+0000","last_unstale":"2024-03-24T11:27:13.345387+0000","last_undegraded":"2024-03-24T11:27:13.345387+0000","last_fullsized":"2024-03-24T11:27:13.345387+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:12.962703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:12.962703+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:21.799976+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":270,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.347522+0000","last_change":"2024-03-24T11:26:37.859404+0000","last_active":"2024-03-24T11:27:13.347522+0000","last_peered":"2024-03-24T11:27:13.347522+0000","last_clean":"2024-03-24T11:27:13.347522+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:27:13.347522+0000","last_undegraded":"2024-03-24T11:27:13.347522+0000","last_fullsized":"2024-03-24T11:27:13.347522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.859286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.859286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:47.488541+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":391,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.378567+0000","last_change":"2024-03-24T11:25:48.800516+0000","last_active":"2024-03-24T11:27:13.378567+0000","last_peered":"2024-03-24T11:27:13.378567+0000","last_clean":"2024-03-24T11:27:13.378567+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:27:13.378567+0000","last_undegraded":"2024-03-24T11:27:13.378567+0000","last_fullsized":"2024-03-24T11:27:13.378567+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:48.800253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:48.800253+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.983999+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":274,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.386234+0000","last_change":"2024-03-24T11:26:46.366056+0000","last_active":"2024-03-24T11:27:13.386234+0000","last_peered":"2024-03-24T11:27:13.386234+0000","last_clean":"2024-03-24T11:27:13.386234+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:27:13.386234+0000","last_undegraded":"2024-03-24T11:27:13.386234+0000","last_fullsized":"2024-03-24T11:27:13.386234+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.365877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.365877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.222226+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":285,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-03-24T11:27:13.345465+0000","last_change":"2024-03-24T11:25:41.084396+0000","last_active":"2024-03-24T11:27:13.345465+0000","last_peered":"2024-03-24T11:27:13.345465+0000","last_clean":"2024-03-24T11:27:13.345465+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:27:13.345465+0000","last_undegraded":"2024-03-24T11:27:13.345465+0000","last_fullsized":"2024-03-24T11:27:13.345465+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:25:41.084261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:41.084261+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:15.721552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5817958424,"num_objects":88776,"num_object_clones":0,"num_object_copies":266328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167304,"num_write_kb":8194497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17471713280,"data_stored":17471701064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81004,"ondisk_log_size":81004,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":492,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":448,"ondisk_log_size":448,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070569,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958812,"kb_used_data":1783188,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91786340,"statfs":{"total":95995035648,"available":93989212160,"internally_reserved":0,"allocated":1825984512,"data_stored":1825685832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":179763659},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364910,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1955980,"kb_used_data":1784580,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91789172,"statfs":{"total":95995035648,"available":93992112128,"internally_reserved":0,"allocated":1827409920,"data_stored":1827127821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2756020,"kb_used_data":2517996,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":90989132,"statfs":{"total":95995035648,"available":93172871168,"internally_reserved":0,"allocated":2578427904,"data_stored":2578105682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2401120,"kb_used_data":2117208,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91344032,"statfs":{"total":95995035648,"available":93536288768,"internally_reserved":0,"allocated":2168020992,"data_stored":2167717947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2402908,"kb_used_data":2155156,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91342244,"statfs":{"total":95995035648,"available":93534457856,"internally_reserved":0,"allocated":2206879744,"data_stored":2206581124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2438580,"kb_used_data":2146348,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91306572,"statfs":{"total":95995035648,"available":93497929728,"internally_reserved":0,"allocated":2197860352,"data_stored":2197537866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986296,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1975524,"kb_used_data":1795804,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91769628,"statfs":{"total":95995035648,"available":93972099072,"internally_reserved":0,"allocated":1838903296,"data_stored":1838596828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659251,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3098072,"kb_used_data":2846160,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90647080,"statfs":{"total":95995035648,"available":92822609920,"internally_reserved":0,"allocated":2914467840,"data_stored":2914173582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2150694912,"data_stored":2150694912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2564358144,"data_stored":2564358144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2192510976,"data_stored":2192506904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2188771328,"data_stored":2188771328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1831862272,"data_stored":1831862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2907181056,"data_stored":2907176984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1814691840,"data_stored":1814691840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1821642752,"data_stored":1821638680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:27:27.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-24T11:27:27.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:27.788 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:27:27.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:27.803 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:27.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: pgmap v341: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 873 op/s 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: 6.b scrub starts 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: 6.b scrub ok 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2876907147' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:27:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:27:27.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:27 smithi181 ceph-mon[33998]: 8.4 scrub ok 2024-03-24T11:27:27.904 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:27.948 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:27.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:28.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:28.103 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:28.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:28.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:28.236 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:28.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:28.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:28.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:28.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:28.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:28.510 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:28.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:28.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:28.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:28.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:28.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:27:28.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2876907147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 5.12 deep-scrub starts 2024-03-24T11:27:28.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[29608]: 5.12 deep-scrub ok 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2876907147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:27:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 5.12 deep-scrub starts 2024-03-24T11:27:28.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:28 smithi179 ceph-mon[37907]: 5.12 deep-scrub ok 2024-03-24T11:27:28.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:28.784 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:28.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:28.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:27:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2876907147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:28.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T11:27:28.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:27:28.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:27:28.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 5.12 deep-scrub starts 2024-03-24T11:27:28.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:28 smithi181 ceph-mon[33998]: 5.12 deep-scrub ok 2024-03-24T11:27:28.927 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:28.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:29.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:29.084 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:29.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:29.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:29.221 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:29.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:29.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:29.363 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:29.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:29.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:29.501 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:29.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:29.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:29.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:27:29.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:27:29.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:27:29.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:27:29.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: pgmap v343: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:27:29.627 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:27:29.628 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:27:29.628 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:27:29.631 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:27:29.631 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:27:29.631 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:27:29.631 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:27:29.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: pgmap v343: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:27:29.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:27:29.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:27:29.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:29 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:27:29.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:29.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:29.721 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:29.722 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:29.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:29.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:29.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:29.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: pgmap v343: 158 pgs: 158 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:27:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:29 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:27:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:29.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:29.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:30.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:30.059 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:30.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:30.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:30.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:30.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:30.292 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:30.330 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:30.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:30.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:30.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:30.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:30.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:30.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:27:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:30.623 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:30.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:30.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:30 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:27:30.665 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:30 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:27:30.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:30 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:27:30.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:30 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:27:30.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:30.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:30.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:30.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:30.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:30.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:30.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:30 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:27:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:30 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:27:30.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:31.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:31.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:31.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:31.175 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:31.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:31.276 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:31.301 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:31.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:31.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:31.443 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:31.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:31.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:31.586 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:31.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:31.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:31.725 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:31.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:31.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:31 smithi181 ceph-mon[33998]: pgmap v345: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-03-24T11:27:31.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:31.869 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:31.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:31 smithi179 ceph-mon[29608]: pgmap v345: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-03-24T11:27:31.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:31 smithi179 ceph-mon[37907]: pgmap v345: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-03-24T11:27:31.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:32.011 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:32.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:32.150 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:32.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:32.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:32.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:32.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:32.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.565 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:32.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 6.15 deep-scrub starts 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 6.15 deep-scrub ok 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 3.1e scrub starts 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 3.1e scrub ok 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:27:32.647 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:32 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:27:32.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:32.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:32.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 6.15 deep-scrub starts 2024-03-24T11:27:32.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 6.15 deep-scrub ok 2024-03-24T11:27:32.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:27:32.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:27:32.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 3.1e scrub starts 2024-03-24T11:27:32.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 3.1e scrub ok 2024-03-24T11:27:32.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 6.15 deep-scrub starts 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 6.15 deep-scrub ok 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 3.1e scrub starts 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 3.1e scrub ok 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:27:32.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:32 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:27:32.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:32.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:32.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:32.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:32.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:33.083 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:33.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:33.225 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:33.272 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:33.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:27:33.334 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:27:33.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:27:33.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:33.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:33.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:33.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:33.554 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:33.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:33.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:33.693 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:33.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:33.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:33.835 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:33.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:33.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:27:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:27:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:27:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: pgmap v346: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 753 op/s 2024-03-24T11:27:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2545822788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: 5.1b deep-scrub starts 2024-03-24T11:27:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: 5.1b deep-scrub ok 2024-03-24T11:27:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:27:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:33 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:27:33.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:27:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:27:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: pgmap v346: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 753 op/s 2024-03-24T11:27:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2545822788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: 5.1b deep-scrub starts 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: 5.1b deep-scrub ok 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: pgmap v346: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 753 op/s 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2545822788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: 5.1b deep-scrub starts 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: 5.1b deep-scrub ok 2024-03-24T11:27:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:27:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:33 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:27:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:33.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:33.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:34.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:34.120 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:34.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:34.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:34.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:34.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:34.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:34.393 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:34.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:34.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:34.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:34.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:34.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:27:34.607 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 4.17 deep-scrub starts 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 4.17 deep-scrub ok 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 5.3 deep-scrub starts 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[29608]: 5.3 deep-scrub ok 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 4.17 deep-scrub starts 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:34.608 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 4.17 deep-scrub ok 2024-03-24T11:27:34.609 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:27:34.609 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:27:34.609 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 5.3 deep-scrub starts 2024-03-24T11:27:34.609 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:34 smithi179 ceph-mon[37907]: 5.3 deep-scrub ok 2024-03-24T11:27:34.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:34.678 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:34.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:34.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:34.814 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:34.823 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:34.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 4.17 deep-scrub starts 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 4.17 deep-scrub ok 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 5.3 deep-scrub starts 2024-03-24T11:27:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:34 smithi181 ceph-mon[33998]: 5.3 deep-scrub ok 2024-03-24T11:27:34.906 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:27:34.906 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:27:34.906 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:27:34.906 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:27:34.906 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:27:34.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:34.953 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:34.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:35.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:35.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:35.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:35.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:35.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:27:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:27:35.228 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:35.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:35.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:35.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:35.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:35.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:35.481 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:27:35.481 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:27:35.482 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:27:35.482 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[37907]: pgmap v347: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-24T11:27:35.482 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:27:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:35.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:35.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:35.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:35.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:35.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:35.733 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:27:35.733 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[29608]: pgmap v347: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-24T11:27:35.733 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:27:35.733 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:27:35.734 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:35 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:27:35.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:35.783 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:35.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:35.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:35.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:35 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:27:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:35 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:27:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:35 smithi181 ceph-mon[33998]: pgmap v347: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-24T11:27:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:35 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:27:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:35 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:27:35.923 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:35.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:36.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:36.056 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:36.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:36.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:27:36.158 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:36.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:36.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:36.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:36.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:36.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:36.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:36.500 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:36.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 1 pgp_num to 11 2024-03-24T11:27:36.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:36.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:36.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:36.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:36.770 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:36.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:36.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:36.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:36 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:27:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:36 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:27:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1460461801' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:27:36.912 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:36.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:36.941 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:36 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:27:36.941 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:36 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:27:36.941 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1460461801' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:27:36.942 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:36 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:27:36.942 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:36 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:27:36.942 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1460461801' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:27:37.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:37.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:37.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:37.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:37.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:37.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:37.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.436 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:37.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:37.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.571 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:37.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:37.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:37.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:37.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:37.849 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:37 smithi181 ceph-mon[33998]: pgmap v348: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-24T11:27:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:37 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1460461801' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T11:27:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:37 smithi181 ceph-mon[33998]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:27:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:37 smithi181 ceph-mon[33998]: 1.4 scrub starts 2024-03-24T11:27:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:37 smithi181 ceph-mon[33998]: 1.4 scrub ok 2024-03-24T11:27:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:37 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:27:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:37 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:27:37.951 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[37907]: pgmap v348: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1460461801' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[37907]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[37907]: 1.4 scrub starts 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[37907]: 1.4 scrub ok 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:27:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[29608]: pgmap v348: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-24T11:27:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1460461801' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T11:27:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[29608]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T11:27:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[29608]: 1.4 scrub starts 2024-03-24T11:27:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[29608]: 1.4 scrub ok 2024-03-24T11:27:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:27:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:37 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:27:37.978 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:37.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:38.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:38.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:38.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:38.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:38.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:38.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:38.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:38.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:38.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:38.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:38.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:38.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:38.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:38.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:38.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 5.2 deep-scrub starts 2024-03-24T11:27:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 5.2 deep-scrub ok 2024-03-24T11:27:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:27:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 6.1f scrub ok 2024-03-24T11:27:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 5.2 deep-scrub starts 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 5.2 deep-scrub ok 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 6.1f scrub ok 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:27:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:27:38.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:27:38.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:27:38.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:38 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:27:38.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:38.791 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:38.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:38.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:38.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 5.2 deep-scrub starts 2024-03-24T11:27:38.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 5.2 deep-scrub ok 2024-03-24T11:27:38.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:27:38.905 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 6.1f scrub ok 2024-03-24T11:27:38.905 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:27:38.905 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:27:38.905 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:27:38.905 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:27:38.905 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:38 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:27:38.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:38.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:39.037 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:39.075 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:39.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:39.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:39.210 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:39.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:39.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:39.336 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:39.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:39.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:39.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:39.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:39.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:39.604 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:39.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:39.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:39.746 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:39.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:39.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:39.876 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:39.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:39 smithi181 ceph-mon[33998]: pgmap v350: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:27:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:39 smithi181 ceph-mon[33998]: 4.11 scrub starts 2024-03-24T11:27:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:39 smithi181 ceph-mon[33998]: 4.11 scrub ok 2024-03-24T11:27:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:39 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:27:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:39 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:27:39.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[29608]: pgmap v350: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:27:39.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[29608]: 4.11 scrub starts 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[29608]: 4.11 scrub ok 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[37907]: pgmap v350: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[37907]: 4.11 scrub starts 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[37907]: 4.11 scrub ok 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:27:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:39 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:27:39.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.015 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:40.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:40.115 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:40.151 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:40.253 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:40.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:40.400 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:40.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.430 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:40.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:40.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:40.566 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:40.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:40.698 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:40.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:40.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:40 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:27:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:40 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:27:40.931 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:40 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:27:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:40 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:27:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:40 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:27:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:40 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:27:40.969 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:40.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:41.071 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:41.109 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:41.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:41.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:41.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:41.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:41.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:41.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:41.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:41.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:41.529 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:41.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:41.631 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:41.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:41.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:41.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:27:41.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:41.772 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[29608]: 1.2 deep-scrub starts 2024-03-24T11:27:41.772 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[29608]: 1.2 deep-scrub ok 2024-03-24T11:27:41.772 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[29608]: pgmap v351: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:27:41.772 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:27:41.772 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:27:41.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[37907]: 1.2 deep-scrub starts 2024-03-24T11:27:41.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[37907]: 1.2 deep-scrub ok 2024-03-24T11:27:41.773 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[37907]: pgmap v351: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:27:41.773 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:27:41.773 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:41 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:27:41.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:41.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:41.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:41 smithi181 ceph-mon[33998]: 1.2 deep-scrub starts 2024-03-24T11:27:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:41 smithi181 ceph-mon[33998]: 1.2 deep-scrub ok 2024-03-24T11:27:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:41 smithi181 ceph-mon[33998]: pgmap v351: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:27:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:41 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:27:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:41 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:27:41.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:41.971 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:41.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:42.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:42.135 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:42.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:42.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:42.287 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:42.297 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:42.310 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:42.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:42.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:42.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:42.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:42.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:42.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:42.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:42.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:42.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:27:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:27:42.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:27:42.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:27:42.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:27:42.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:42 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:42.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:42.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:42.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:42.866 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:42.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:42 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:27:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:42 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:27:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:42 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:27:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:42 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:27:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:42 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:27:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:42 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:27:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:42 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:42.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:42.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:42.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:43.100 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:43.138 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:43.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:43.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:43.273 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:43.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:43.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:43.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:43.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:43.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:43.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:43.552 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:43.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:43.560 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:27:43.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: pgmap v352: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2088734539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:27:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:27:43.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:43 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:27:43.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:43.691 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:43.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:27:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:27:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: pgmap v352: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:27:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2088734539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:27:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:27:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: pgmap v352: 158 pgs: 158 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2088734539' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:27:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:27:43.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:27:43.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:43 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:27:43.793 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:43.831 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:43.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:43.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:43.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:43.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:44.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:44.111 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:44.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:44.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:44.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:44.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:44.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:44.387 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:44.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:44.489 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:44.532 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:44.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:44.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:44.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:44.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:44.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:44.813 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:44.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:44 smithi181 ceph-mon[33998]: 8.b scrub starts 2024-03-24T11:27:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:44 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:44 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:27:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:44 smithi181 ceph-mon[33998]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:27:44.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:44.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:44.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[37907]: 8.b scrub starts 2024-03-24T11:27:44.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:44.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:44.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:27:44.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[37907]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:27:44.949 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[29608]: 8.b scrub starts 2024-03-24T11:27:44.949 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:44.949 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:44.949 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:27:44.949 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:44 smithi179 ceph-mon[29608]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T11:27:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:44.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:45.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:45.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:45.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:45.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:45.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:27:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:27:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:27:45.230 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:45.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:45.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:45.367 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:45.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:45.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:45.503 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:45.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:45.605 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:45.641 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:45.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:45 smithi179 ceph-mon[29608]: pgmap v354: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-24T11:27:45.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:45 smithi179 ceph-mon[37907]: pgmap v354: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-24T11:27:45.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:45.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:45.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:45.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:45 smithi181 ceph-mon[33998]: pgmap v354: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-24T11:27:45.914 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:45.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:46.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:46.055 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:46.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:46.155 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:27:46.155 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:27:46.155 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-24T11:27:46.155 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:27:46.155 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:27:46.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-24T11:27:46.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:46.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:46.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:46.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:46.337 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:46.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:46.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:46.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:46.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:46.560 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:27:46.588 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:46.634 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:46.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:46.735 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:46.769 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:46.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:46.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:46.905 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:46.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:47.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:47.048 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:47.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:47.150 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:47.165 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:27:47.188 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:47.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:47.216 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:27:47.254 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:27:47.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:47.293 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:27:47.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:47.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:47.331 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:27:47.371 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:27:47.414 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:27:47.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:47.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:27:47.454 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:27:47.457 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:27:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:47.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:47.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:47.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:47.601 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:47.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:47.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:47.731 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:47 smithi179 ceph-mon[29608]: pgmap v355: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:27:47.732 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4277794937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:47.732 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:47 smithi179 ceph-mon[37907]: pgmap v355: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:27:47.732 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4277794937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:47.732 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:27:47.732 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:27:47.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:47.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:47.819 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:47 smithi181 ceph-mon[33998]: pgmap v355: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:27:47.819 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4277794937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:47.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:47.877 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:47.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:47.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:48.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:48.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:48.120 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:48.158 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:48.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:48.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:48.301 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:48.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:48.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:48.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:48.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:48.545 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:48.578 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:48.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:48.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:48.714 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:48.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:48.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:48.822 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:27:48.822 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T11:27:48.822 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:27:48.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:48.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:48.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:49.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:49.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:49.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:49.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:49.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:49.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:49.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:49.205 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:49.205 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:27:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:27:49.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:49.257 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:27:49.299 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:49.300 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:49.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:49.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:49.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:49.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:49.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:49.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:49.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:49.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:49.714 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:49.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:49.817 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:49.859 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:49.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:49.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:50.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:50.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:50.102 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:50.141 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:50.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:50.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:49 smithi181 ceph-mon[33998]: pgmap v356: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:27:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:49 smithi179 ceph-mon[37907]: pgmap v356: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:27:50.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:49 smithi179 ceph-mon[29608]: pgmap v356: 158 pgs: 158 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:27:50.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:50.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:50.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:50.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:50.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:50.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:50.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:27:50.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:50.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:50.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:50.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:50.726 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:50.759 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:50.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:50.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:50.907 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:50.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:51.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:51.046 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:51.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:51.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:51.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:50 smithi181 ceph-mon[33998]: pgmap v357: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T11:27:51.157 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:50 smithi179 ceph-mon[29608]: pgmap v357: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T11:27:51.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:50 smithi179 ceph-mon[37907]: pgmap v357: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T11:27:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:51.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:51.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:51.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:51.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:51.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:51.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:51.454 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:51.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:51.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:51.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:51.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:51.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:51.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:51.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:51.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:51.867 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:51.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:51.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:52.009 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:52.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:52.141 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:52.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:52.287 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:52.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:52.429 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:52.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:52.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:52.665 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:52.703 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:52.767 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:52.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:52.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:52.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:52.986 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:52.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:53.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:53.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:53.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:53.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:53.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:53.259 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:53.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:53.271 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:27:53.344 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:53 smithi179 ceph-mon[29608]: pgmap v358: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T11:27:53.344 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2420615139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:53.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:53 smithi179 ceph-mon[37907]: pgmap v358: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T11:27:53.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2420615139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:53.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:53.399 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:53.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:53.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:53.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:53.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:53.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:53.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:53 smithi181 ceph-mon[33998]: pgmap v358: 158 pgs: 158 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T11:27:53.538 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2420615139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:27:53.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:53.683 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:53.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:53.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:53.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:53.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:53.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:27:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:27:53.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:53.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:53.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:54.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:54.093 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:54.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:54.166 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:27:54.166 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:27:54.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:27:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:54.231 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:54.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:27:54.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:54.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:54.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:54.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:54.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2420615139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:54.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[29608]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:27:54.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2420615139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[37907]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:27:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:54.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:54.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:54.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:54.606 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:27:54.606 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:27:54.606 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-24T11:27:54.606 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-24T11:27:54.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:27:54.608 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:54 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:27:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2420615139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:27:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:54 smithi181 ceph-mon[33998]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T11:27:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:27:54.654 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:54.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:54.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:54.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:54.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:54.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:54.941 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:54.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:55.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:55.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:55.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:55.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:55.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:55.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:27:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:27:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:27:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:55.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:55.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:55.361 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:55.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:55.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:55.501 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:55.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:55.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:55.639 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:55.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:55 smithi181 ceph-mon[33998]: pgmap v360: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:27:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:55 smithi179 ceph-mon[29608]: pgmap v360: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:27:55.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:55 smithi179 ceph-mon[37907]: pgmap v360: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:27:55.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:55.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:55.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:55.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:55.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:55.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:56.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:56.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:56.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.176 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:56.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:56.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:56.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:56.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.447 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:56.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:56.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:56.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:56.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:56.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:56.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:27:56.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:56.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:56.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:56.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:56.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:57.094 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:57.134 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:57.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:57.236 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:27:57.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:57.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:57.266 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:57.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:57.350 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:27:57.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:57.394 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:27:57.416 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:57.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:57.432 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:27:57.473 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:27:57.509 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:27:57.510 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:57 smithi179 ceph-mon[29608]: pgmap v361: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:27:57.510 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:57 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:27:57.510 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:57 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:27:57.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:57 smithi179 ceph-mon[37907]: pgmap v361: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:27:57.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:57 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:27:57.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:57 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:27:57.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:57.554 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:27:57.558 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:57.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:57.595 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:27:57.639 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:27:57.646 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:27:57.646 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:27:57.647 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:27:57.647 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:27:57.647 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:27:57.647 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:27:57.647 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:27:57.647 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:27:57.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:57 smithi181 ceph-mon[33998]: pgmap v361: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:27:57.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:57 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:27:57.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:57 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:27:57.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:57.693 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:57.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:57.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:57.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:57.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:57.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:57.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:27:57.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:57.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:58.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:58.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:58.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:58.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:58.241 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:58.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:58.296 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:58 smithi179 ceph-mon[29608]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:27:58.296 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3318920486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:58.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:58.377 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:58.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:58.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:58.516 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:58.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:58.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:58.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:58 smithi181 ceph-mon[33998]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:27:58.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3318920486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:58.659 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:58.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:58.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:58 smithi179 ceph-mon[37907]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T11:27:58.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3318920486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:27:58.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:27:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:58.824 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:58.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:27:58.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:27:58.964 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:58.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:27:59.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:59.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:59.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:59.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:27:59.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:27:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:59.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:27:59.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:59.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:59.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:27:59.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:27:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:59.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:59.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:27:59.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:27:59.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:27:59 smithi181 ceph-mon[33998]: pgmap v363: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s 2024-03-24T11:27:59.659 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:59.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:27:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:27:59 smithi179 ceph-mon[29608]: pgmap v363: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s 2024-03-24T11:27:59.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:27:59 smithi179 ceph-mon[37907]: pgmap v363: 158 pgs: 158 active+clean; 2.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s 2024-03-24T11:27:59.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:27:59.787 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:59.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:27:59.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:27:59.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:27:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:27:59.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:00.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:00.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:00.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:00.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:00.196 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:00.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:00.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:00.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:00.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:00.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:00.465 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:00.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:00.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:00.567 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:00.606 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:00.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:00.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:00.754 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:00.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:00.775 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:00.778 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:00.854 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:00.905 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:00.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:01.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:01.047 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:01.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:01.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:01.182 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:01.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:01.258 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:28:01.261 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:28:01.275 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:28:01.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:01.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:01.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:01.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:01.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:01.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:01.552 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:01.599 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:01.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:01.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:01 smithi181 ceph-mon[33998]: pgmap v364: 158 pgs: 158 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-03-24T11:28:01.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:01 smithi179 ceph-mon[29608]: pgmap v364: 158 pgs: 158 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-03-24T11:28:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:01 smithi179 ceph-mon[37907]: pgmap v364: 158 pgs: 158 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-03-24T11:28:01.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:01.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:01.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:01.875 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:01.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:01.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:02.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:02.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.151 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:02.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:02.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:02.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.277 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:02.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:02.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:02.417 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:02.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:02.552 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:02.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:02 smithi181 ceph-mon[33998]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:28:02.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:02.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:02 smithi179 ceph-mon[29608]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:28:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:02 smithi179 ceph-mon[37907]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:28:02.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:02.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:02.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:02.968 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:02.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:03.069 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:03.097 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:03.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:03.199 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:03.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:03.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:03.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:03.373 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:03.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:03.475 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:03.513 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:03.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:03.614 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:03.615 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:03 smithi179 ceph-mon[29608]: pgmap v365: 158 pgs: 158 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 573 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:28:03.615 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:03.616 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:03 smithi179 ceph-mon[37907]: pgmap v365: 158 pgs: 158 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 573 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:28:03.616 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:03.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:03.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:03.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:03 smithi181 ceph-mon[33998]: pgmap v365: 158 pgs: 158 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 573 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:28:03.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:03.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:28:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:28:03.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:03.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:03.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:03.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:03.919 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:03.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:04.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:04.059 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:04.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:04.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:04.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:04.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:04.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:04.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:04.337 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:04.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:04.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:04.477 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:04.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:04.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:04.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:04.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:04.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:04.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:04.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:04.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:04.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:04.812 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-03-24T11:28:00.192688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2759334262,"num_objects":42294,"num_object_clones":0,"num_object_copies":126882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42294,"num_whiteouts":0,"num_read":1259,"num_read_kb":1011,"num_write":214420,"num_write_kb":8194542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83028,"ondisk_log_size":83028,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10313960,"kb_used_data":8208072,"kb_used_omap":12,"kb_used_meta":2105459,"kb_avail":739647256,"statfs":{"total":767960285184,"available":757398790144,"internally_reserved":0,"allocated":8405065728,"data_stored":8402441082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2155990577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-780599296,"num_objects":-11911,"num_object_clones":0,"num_object_copies":-35733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11911,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11911,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002441"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284528+0000","last_change":"2024-03-24T11:26:32.919289+0000","last_active":"2024-03-24T11:27:57.284528+0000","last_peered":"2024-03-24T11:27:57.284528+0000","last_clean":"2024-03-24T11:27:57.284528+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:27:57.284528+0000","last_undegraded":"2024-03-24T11:27:57.284528+0000","last_fullsized":"2024-03-24T11:27:57.284528+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.919158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.919158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.078522+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295196+0000","last_change":"2024-03-24T11:27:17.030232+0000","last_active":"2024-03-24T11:27:57.295196+0000","last_peered":"2024-03-24T11:27:57.295196+0000","last_clean":"2024-03-24T11:27:57.295196+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:27:57.295196+0000","last_undegraded":"2024-03-24T11:27:57.295196+0000","last_fullsized":"2024-03-24T11:27:57.295196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.030061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.030061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:23.838953+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296842+0000","last_change":"2024-03-24T11:27:33.388210+0000","last_active":"2024-03-24T11:27:57.296842+0000","last_peered":"2024-03-24T11:27:57.296842+0000","last_clean":"2024-03-24T11:27:57.296842+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:27:57.296842+0000","last_undegraded":"2024-03-24T11:27:57.296842+0000","last_fullsized":"2024-03-24T11:27:57.296842+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:33.388096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:33.388096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:48.767655+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":322,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284025+0000","last_change":"2024-03-24T11:26:40.754563+0000","last_active":"2024-03-24T11:27:57.284025+0000","last_peered":"2024-03-24T11:27:57.284025+0000","last_clean":"2024-03-24T11:27:57.284025+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:27:57.284025+0000","last_undegraded":"2024-03-24T11:27:57.284025+0000","last_fullsized":"2024-03-24T11:27:57.284025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:26:40.754425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:40.754425+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120495+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":266,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311740+0000","last_change":"2024-03-24T11:25:38.404738+0000","last_active":"2024-03-24T11:27:57.311740+0000","last_peered":"2024-03-24T11:27:57.311740+0000","last_clean":"2024-03-24T11:27:57.311740+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:27:57.311740+0000","last_undegraded":"2024-03-24T11:27:57.311740+0000","last_fullsized":"2024-03-24T11:27:57.311740+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:38.404511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.404511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448022+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":286,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284115+0000","last_change":"2024-03-24T11:27:11.705436+0000","last_active":"2024-03-24T11:27:57.284115+0000","last_peered":"2024-03-24T11:27:57.284115+0000","last_clean":"2024-03-24T11:27:57.284115+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:27:57.284115+0000","last_undegraded":"2024-03-24T11:27:57.284115+0000","last_fullsized":"2024-03-24T11:27:57.284115+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:11.705334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.705334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.113548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.312924+0000","last_change":"2024-03-24T11:27:31.498691+0000","last_active":"2024-03-24T11:27:57.312924+0000","last_peered":"2024-03-24T11:27:57.312924+0000","last_clean":"2024-03-24T11:27:57.312924+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:27:57.312924+0000","last_undegraded":"2024-03-24T11:27:57.312924+0000","last_fullsized":"2024-03-24T11:27:57.312924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:31.498583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.498583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:00.354290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":299,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311353+0000","last_change":"2024-03-24T11:26:29.288367+0000","last_active":"2024-03-24T11:27:57.311353+0000","last_peered":"2024-03-24T11:27:57.311353+0000","last_clean":"2024-03-24T11:27:57.311353+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:27:57.311353+0000","last_undegraded":"2024-03-24T11:27:57.311353+0000","last_fullsized":"2024-03-24T11:27:57.311353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:26:29.288223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.288223+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448132+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295816+0000","last_change":"2024-03-24T11:27:14.464091+0000","last_active":"2024-03-24T11:27:57.295816+0000","last_peered":"2024-03-24T11:27:57.295816+0000","last_clean":"2024-03-24T11:27:57.295816+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:27:57.295816+0000","last_undegraded":"2024-03-24T11:27:57.295816+0000","last_fullsized":"2024-03-24T11:27:57.295816+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:14.463973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:14.463973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:16.251800+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292994+0000","last_change":"2024-03-24T11:27:32.894325+0000","last_active":"2024-03-24T11:27:57.292994+0000","last_peered":"2024-03-24T11:27:57.292994+0000","last_clean":"2024-03-24T11:27:57.292994+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:27:57.292994+0000","last_undegraded":"2024-03-24T11:27:57.292994+0000","last_fullsized":"2024-03-24T11:27:57.292994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:27:32.894112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:01.062230+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292851+0000","last_change":"2024-03-24T11:25:46.067948+0000","last_active":"2024-03-24T11:27:57.292851+0000","last_peered":"2024-03-24T11:27:57.292851+0000","last_clean":"2024-03-24T11:27:57.292851+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:27:57.292851+0000","last_undegraded":"2024-03-24T11:27:57.292851+0000","last_fullsized":"2024-03-24T11:27:57.292851+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:25:46.067824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":306,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284796+0000","last_change":"2024-03-24T11:27:29.941329+0000","last_active":"2024-03-24T11:27:57.284796+0000","last_peered":"2024-03-24T11:27:57.284796+0000","last_clean":"2024-03-24T11:27:57.284796+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:27:57.284796+0000","last_undegraded":"2024-03-24T11:27:57.284796+0000","last_fullsized":"2024-03-24T11:27:57.284796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:27:29.941182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:29.941182+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:44.084844+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295148+0000","last_change":"2024-03-24T11:26:42.053183+0000","last_active":"2024-03-24T11:27:57.295148+0000","last_peered":"2024-03-24T11:27:57.295148+0000","last_clean":"2024-03-24T11:27:57.295148+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:27:57.295148+0000","last_undegraded":"2024-03-24T11:27:57.295148+0000","last_fullsized":"2024-03-24T11:27:57.295148+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.052777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.052777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790375+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":286,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292518+0000","last_change":"2024-03-24T11:27:34.094930+0000","last_active":"2024-03-24T11:27:57.292518+0000","last_peered":"2024-03-24T11:27:57.292518+0000","last_clean":"2024-03-24T11:27:57.292518+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:27:57.292518+0000","last_undegraded":"2024-03-24T11:27:57.292518+0000","last_fullsized":"2024-03-24T11:27:57.292518+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:34.094788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:27:34.094788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:53.948199+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294402+0000","last_change":"2024-03-24T11:26:43.090579+0000","last_active":"2024-03-24T11:27:57.294402+0000","last_peered":"2024-03-24T11:27:57.294402+0000","last_clean":"2024-03-24T11:27:57.294402+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:27:57.294402+0000","last_undegraded":"2024-03-24T11:27:57.294402+0000","last_fullsized":"2024-03-24T11:27:57.294402+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:43.090367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.090367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:06.561184+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":300,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284977+0000","last_change":"2024-03-24T11:26:47.926021+0000","last_active":"2024-03-24T11:27:57.284977+0000","last_peered":"2024-03-24T11:27:57.284977+0000","last_clean":"2024-03-24T11:27:57.284977+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:27:57.284977+0000","last_undegraded":"2024-03-24T11:27:57.284977+0000","last_fullsized":"2024-03-24T11:27:57.284977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-24T11:26:47.925773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.925773+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120594+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":283,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286011+0000","last_change":"2024-03-24T11:26:27.996936+0000","last_active":"2024-03-24T11:27:57.286011+0000","last_peered":"2024-03-24T11:27:57.286011+0000","last_clean":"2024-03-24T11:27:57.286011+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:27:57.286011+0000","last_undegraded":"2024-03-24T11:27:57.286011+0000","last_fullsized":"2024-03-24T11:27:57.286011+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:27.996815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.996815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120551+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293662+0000","last_change":"2024-03-24T11:27:18.078096+0000","last_active":"2024-03-24T11:27:57.293662+0000","last_peered":"2024-03-24T11:27:57.293662+0000","last_clean":"2024-03-24T11:27:57.293662+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:27:57.293662+0000","last_undegraded":"2024-03-24T11:27:57.293662+0000","last_fullsized":"2024-03-24T11:27:57.293662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:18.077961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.077961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:38.436426+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296290+0000","last_change":"2024-03-24T11:26:32.451559+0000","last_active":"2024-03-24T11:27:57.296290+0000","last_peered":"2024-03-24T11:27:57.296290+0000","last_clean":"2024-03-24T11:27:57.296290+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:27:57.296290+0000","last_undegraded":"2024-03-24T11:27:57.296290+0000","last_fullsized":"2024-03-24T11:27:57.296290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.451446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.451446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.270251+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":323,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292920+0000","last_change":"2024-03-24T11:27:22.975616+0000","last_active":"2024-03-24T11:27:57.292920+0000","last_peered":"2024-03-24T11:27:57.292920+0000","last_clean":"2024-03-24T11:27:57.292920+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:27:57.292920+0000","last_undegraded":"2024-03-24T11:27:57.292920+0000","last_fullsized":"2024-03-24T11:27:57.292920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:27:22.975468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:22.975468+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.206985+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":275,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.315939+0000","last_change":"2024-03-24T11:27:37.602861+0000","last_active":"2024-03-24T11:27:57.315939+0000","last_peered":"2024-03-24T11:27:57.315939+0000","last_clean":"2024-03-24T11:27:57.315939+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:27:57.315939+0000","last_undegraded":"2024-03-24T11:27:57.315939+0000","last_fullsized":"2024-03-24T11:27:57.315939+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:37.602750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:37.602750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.113684+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":286,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284762+0000","last_change":"2024-03-24T11:26:26.941914+0000","last_active":"2024-03-24T11:27:57.284762+0000","last_peered":"2024-03-24T11:27:57.284762+0000","last_clean":"2024-03-24T11:27:57.284762+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:27:57.284762+0000","last_undegraded":"2024-03-24T11:27:57.284762+0000","last_fullsized":"2024-03-24T11:27:57.284762+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.941809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.941809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120406+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284851+0000","last_change":"2024-03-24T11:27:28.920285+0000","last_active":"2024-03-24T11:27:57.284851+0000","last_peered":"2024-03-24T11:27:57.284851+0000","last_clean":"2024-03-24T11:27:57.284851+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:27:57.284851+0000","last_undegraded":"2024-03-24T11:27:57.284851+0000","last_fullsized":"2024-03-24T11:27:57.284851+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:28.920173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:28.920173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.885240+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":297,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.312472+0000","last_change":"2024-03-24T11:25:39.305828+0000","last_active":"2024-03-24T11:27:57.312472+0000","last_peered":"2024-03-24T11:27:57.312472+0000","last_clean":"2024-03-24T11:27:57.312472+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:27:57.312472+0000","last_undegraded":"2024-03-24T11:27:57.312472+0000","last_fullsized":"2024-03-24T11:27:57.312472+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:25:39.305712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.305712+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448057+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":284,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293410+0000","last_change":"2024-03-24T11:27:17.910132+0000","last_active":"2024-03-24T11:27:57.293410+0000","last_peered":"2024-03-24T11:27:57.293410+0000","last_clean":"2024-03-24T11:27:57.293410+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:27:57.293410+0000","last_undegraded":"2024-03-24T11:27:57.293410+0000","last_fullsized":"2024-03-24T11:27:57.293410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:27:17.910050+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.910050+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:26.378109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":419,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.287532+0000","last_change":"2024-03-24T11:27:18.401734+0000","last_active":"2024-03-24T11:27:57.287532+0000","last_peered":"2024-03-24T11:27:57.287532+0000","last_clean":"2024-03-24T11:27:57.287532+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:27:57.287532+0000","last_undegraded":"2024-03-24T11:27:57.287532+0000","last_fullsized":"2024-03-24T11:27:57.287532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:27:18.401578+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:23:53.129429+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.401578+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:27.954587+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":303,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295288+0000","last_change":"2024-03-24T11:26:26.669460+0000","last_active":"2024-03-24T11:27:57.295288+0000","last_peered":"2024-03-24T11:27:57.295288+0000","last_clean":"2024-03-24T11:27:57.295288+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:27:57.295288+0000","last_undegraded":"2024-03-24T11:27:57.295288+0000","last_fullsized":"2024-03-24T11:27:57.295288+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:26:26.669216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.669216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":314,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296094+0000","last_change":"2024-03-24T11:27:23.363240+0000","last_active":"2024-03-24T11:27:57.296094+0000","last_peered":"2024-03-24T11:27:57.296094+0000","last_clean":"2024-03-24T11:27:57.296094+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:27:57.296094+0000","last_undegraded":"2024-03-24T11:27:57.296094+0000","last_fullsized":"2024-03-24T11:27:57.296094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:27:23.362869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.362869+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:41.871796+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":290,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.289053+0000","last_change":"2024-03-24T11:27:24.109149+0000","last_active":"2024-03-24T11:27:57.289053+0000","last_peered":"2024-03-24T11:27:57.289053+0000","last_clean":"2024-03-24T11:27:57.289053+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:27:57.289053+0000","last_undegraded":"2024-03-24T11:27:57.289053+0000","last_fullsized":"2024-03-24T11:27:57.289053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:24.108894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:58.788469+0000","last_clean_scrub_stamp":"2024-03-24T11:27:24.108894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:27.613843+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296338+0000","last_change":"2024-03-24T11:26:31.491413+0000","last_active":"2024-03-24T11:27:57.296338+0000","last_peered":"2024-03-24T11:27:57.296338+0000","last_clean":"2024-03-24T11:27:57.296338+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:27:57.296338+0000","last_undegraded":"2024-03-24T11:27:57.296338+0000","last_fullsized":"2024-03-24T11:27:57.296338+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:31.491292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:31.491292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.469946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":286,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.310525+0000","last_change":"2024-03-24T11:26:26.240418+0000","last_active":"2024-03-24T11:27:57.310525+0000","last_peered":"2024-03-24T11:27:57.310525+0000","last_clean":"2024-03-24T11:27:57.310525+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:27:57.310525+0000","last_undegraded":"2024-03-24T11:27:57.310525+0000","last_fullsized":"2024-03-24T11:27:57.310525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.240302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.240302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448218+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":331,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.287438+0000","last_change":"2024-03-24T11:27:23.466973+0000","last_active":"2024-03-24T11:27:57.287438+0000","last_peered":"2024-03-24T11:27:57.287438+0000","last_clean":"2024-03-24T11:27:57.287438+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:27:57.287438+0000","last_undegraded":"2024-03-24T11:27:57.287438+0000","last_fullsized":"2024-03-24T11:27:57.287438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:27:23.466824+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.466824+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.025440+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":285,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292425+0000","last_change":"2024-03-24T11:27:20.015059+0000","last_active":"2024-03-24T11:27:57.292425+0000","last_peered":"2024-03-24T11:27:57.292425+0000","last_clean":"2024-03-24T11:27:57.292425+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:27:57.292425+0000","last_undegraded":"2024-03-24T11:27:57.292425+0000","last_fullsized":"2024-03-24T11:27:57.292425+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:20.014981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:20.014981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.746829+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286909+0000","last_change":"2024-03-24T11:26:47.560502+0000","last_active":"2024-03-24T11:27:57.286909+0000","last_peered":"2024-03-24T11:27:57.286909+0000","last_clean":"2024-03-24T11:27:57.286909+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:27:57.286909+0000","last_undegraded":"2024-03-24T11:27:57.286909+0000","last_fullsized":"2024-03-24T11:27:57.286909+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:47.560381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.560381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.212240+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":302,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.297036+0000","last_change":"2024-03-24T11:27:43.240516+0000","last_active":"2024-03-24T11:27:57.297036+0000","last_peered":"2024-03-24T11:27:57.297036+0000","last_clean":"2024-03-24T11:27:57.297036+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:27:57.297036+0000","last_undegraded":"2024-03-24T11:27:57.297036+0000","last_fullsized":"2024-03-24T11:27:57.297036+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:43.240401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:43.240401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.243033+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"87'6","reported_seq":290,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.310607+0000","last_change":"2024-03-24T11:27:13.458460+0000","last_active":"2024-03-24T11:27:57.310607+0000","last_peered":"2024-03-24T11:27:57.310607+0000","last_clean":"2024-03-24T11:27:57.310607+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:27:57.310607+0000","last_undegraded":"2024-03-24T11:27:57.310607+0000","last_fullsized":"2024-03-24T11:27:57.310607+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'6","last_scrub_stamp":"2024-03-24T11:27:13.458330+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:27:13.458330+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.492165+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311923+0000","last_change":"2024-03-24T11:27:41.544856+0000","last_active":"2024-03-24T11:27:57.311923+0000","last_peered":"2024-03-24T11:27:57.311923+0000","last_clean":"2024-03-24T11:27:57.311923+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:27:57.311923+0000","last_undegraded":"2024-03-24T11:27:57.311923+0000","last_fullsized":"2024-03-24T11:27:57.311923+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:41.544721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:41.544721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:03.082435+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":286,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292643+0000","last_change":"2024-03-24T11:27:33.054622+0000","last_active":"2024-03-24T11:27:57.292643+0000","last_peered":"2024-03-24T11:27:57.292643+0000","last_clean":"2024-03-24T11:27:57.292643+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:27:57.292643+0000","last_undegraded":"2024-03-24T11:27:57.292643+0000","last_fullsized":"2024-03-24T11:27:57.292643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:33.054503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:33.054503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:02.259387+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292806+0000","last_change":"2024-03-24T11:26:30.097875+0000","last_active":"2024-03-24T11:27:57.292806+0000","last_peered":"2024-03-24T11:27:57.292806+0000","last_clean":"2024-03-24T11:27:57.292806+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:27:57.292806+0000","last_undegraded":"2024-03-24T11:27:57.292806+0000","last_fullsized":"2024-03-24T11:27:57.292806+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:30.097693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:30.097693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790326+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":301,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.310742+0000","last_change":"2024-03-24T11:27:38.496836+0000","last_active":"2024-03-24T11:27:57.310742+0000","last_peered":"2024-03-24T11:27:57.310742+0000","last_clean":"2024-03-24T11:27:57.310742+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:27:57.310742+0000","last_undegraded":"2024-03-24T11:27:57.310742+0000","last_fullsized":"2024-03-24T11:27:57.310742+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:27:38.496690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:38.496690+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:52.289415+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295522+0000","last_change":"2024-03-24T11:27:26.375854+0000","last_active":"2024-03-24T11:27:57.295522+0000","last_peered":"2024-03-24T11:27:57.295522+0000","last_clean":"2024-03-24T11:27:57.295522+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:27:57.295522+0000","last_undegraded":"2024-03-24T11:27:57.295522+0000","last_fullsized":"2024-03-24T11:27:57.295522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:26.375744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:26.375744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.913832+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":278,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311652+0000","last_change":"2024-03-24T11:27:12.419817+0000","last_active":"2024-03-24T11:27:57.311652+0000","last_peered":"2024-03-24T11:27:57.311652+0000","last_clean":"2024-03-24T11:27:57.311652+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:27:57.311652+0000","last_undegraded":"2024-03-24T11:27:57.311652+0000","last_fullsized":"2024-03-24T11:27:57.311652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:12.419688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:12.419688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.788280+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293196+0000","last_change":"2024-03-24T11:27:32.055899+0000","last_active":"2024-03-24T11:27:57.293196+0000","last_peered":"2024-03-24T11:27:57.293196+0000","last_clean":"2024-03-24T11:27:57.293196+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:27:57.293196+0000","last_undegraded":"2024-03-24T11:27:57.293196+0000","last_fullsized":"2024-03-24T11:27:57.293196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:32.055790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:32.055790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:00.944756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":314,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293978+0000","last_change":"2024-03-24T11:27:19.050325+0000","last_active":"2024-03-24T11:27:57.293978+0000","last_peered":"2024-03-24T11:27:57.293978+0000","last_clean":"2024-03-24T11:27:57.293978+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:27:57.293978+0000","last_undegraded":"2024-03-24T11:27:57.293978+0000","last_fullsized":"2024-03-24T11:27:57.293978+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:27:19.050122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:19.050122+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.462032+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":274,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.310433+0000","last_change":"2024-03-24T11:26:33.238031+0000","last_active":"2024-03-24T11:27:57.310433+0000","last_peered":"2024-03-24T11:27:57.310433+0000","last_clean":"2024-03-24T11:27:57.310433+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:27:57.310433+0000","last_undegraded":"2024-03-24T11:27:57.310433+0000","last_fullsized":"2024-03-24T11:27:57.310433+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.237864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.237864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448249+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":302,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293796+0000","last_change":"2024-03-24T11:27:28.018429+0000","last_active":"2024-03-24T11:27:57.293796+0000","last_peered":"2024-03-24T11:27:57.293796+0000","last_clean":"2024-03-24T11:27:57.293796+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:27:57.293796+0000","last_undegraded":"2024-03-24T11:27:57.293796+0000","last_fullsized":"2024-03-24T11:27:57.293796+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:27:28.018183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:49.907474+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284662+0000","last_change":"2024-03-24T11:26:33.874445+0000","last_active":"2024-03-24T11:27:57.284662+0000","last_peered":"2024-03-24T11:27:57.284662+0000","last_clean":"2024-03-24T11:27:57.284662+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:27:57.284662+0000","last_undegraded":"2024-03-24T11:27:57.284662+0000","last_fullsized":"2024-03-24T11:27:57.284662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.874331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.874331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:01.789419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"129'16","reported_seq":322,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.285285+0000","last_change":"2024-03-24T11:27:39.010991+0000","last_active":"2024-03-24T11:27:57.285285+0000","last_peered":"2024-03-24T11:27:57.285285+0000","last_clean":"2024-03-24T11:27:57.285285+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:27:57.285285+0000","last_undegraded":"2024-03-24T11:27:57.285285+0000","last_fullsized":"2024-03-24T11:27:57.285285+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'16","last_scrub_stamp":"2024-03-24T11:27:39.010853+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:27:39.010853+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:43.107310+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295793+0000","last_change":"2024-03-24T11:27:31.452484+0000","last_active":"2024-03-24T11:27:57.295793+0000","last_peered":"2024-03-24T11:27:57.295793+0000","last_clean":"2024-03-24T11:27:57.295793+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:27:57.295793+0000","last_undegraded":"2024-03-24T11:27:57.295793+0000","last_fullsized":"2024-03-24T11:27:57.295793+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:31.452376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.452376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:47.520811+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":403,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292937+0000","last_change":"2024-03-24T11:25:25.010482+0000","last_active":"2024-03-24T11:27:57.292937+0000","last_peered":"2024-03-24T11:27:57.292937+0000","last_clean":"2024-03-24T11:27:57.292937+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:27:57.292937+0000","last_undegraded":"2024-03-24T11:27:57.292937+0000","last_fullsized":"2024-03-24T11:27:57.292937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:25.010351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:25.010351+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292620+0000","last_change":"2024-03-24T11:27:35.074782+0000","last_active":"2024-03-24T11:27:57.292620+0000","last_peered":"2024-03-24T11:27:57.292620+0000","last_clean":"2024-03-24T11:27:57.292620+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:27:57.292620+0000","last_undegraded":"2024-03-24T11:27:57.292620+0000","last_fullsized":"2024-03-24T11:27:57.292620+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:35.074657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:35.074657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:35.542279+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":297,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.312417+0000","last_change":"2024-03-24T11:27:24.533619+0000","last_active":"2024-03-24T11:27:57.312417+0000","last_peered":"2024-03-24T11:27:57.312417+0000","last_clean":"2024-03-24T11:27:57.312417+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:27:57.312417+0000","last_undegraded":"2024-03-24T11:27:57.312417+0000","last_fullsized":"2024-03-24T11:27:57.312417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:27:24.533484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:24.533484+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.684360+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286002+0000","last_change":"2024-03-24T11:26:39.812393+0000","last_active":"2024-03-24T11:27:57.286002+0000","last_peered":"2024-03-24T11:27:57.286002+0000","last_clean":"2024-03-24T11:27:57.286002+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:27:57.286002+0000","last_undegraded":"2024-03-24T11:27:57.286002+0000","last_fullsized":"2024-03-24T11:27:57.286002+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.812259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.728524+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":395,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.312599+0000","last_change":"2024-03-24T11:25:49.322508+0000","last_active":"2024-03-24T11:27:57.312599+0000","last_peered":"2024-03-24T11:27:57.312599+0000","last_clean":"2024-03-24T11:27:57.312599+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:27:57.312599+0000","last_undegraded":"2024-03-24T11:27:57.312599+0000","last_fullsized":"2024-03-24T11:27:57.312599+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:49.322383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.322383+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448096+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295934+0000","last_change":"2024-03-24T11:26:46.640485+0000","last_active":"2024-03-24T11:27:57.295934+0000","last_peered":"2024-03-24T11:27:57.295934+0000","last_clean":"2024-03-24T11:27:57.295934+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:27:57.295934+0000","last_undegraded":"2024-03-24T11:27:57.295934+0000","last_fullsized":"2024-03-24T11:27:57.295934+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.640304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.327035+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":314,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293159+0000","last_change":"2024-03-24T11:27:39.856707+0000","last_active":"2024-03-24T11:27:57.293159+0000","last_peered":"2024-03-24T11:27:57.293159+0000","last_clean":"2024-03-24T11:27:57.293159+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:27:57.293159+0000","last_undegraded":"2024-03-24T11:27:57.293159+0000","last_fullsized":"2024-03-24T11:27:57.293159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:27:39.856583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:39.856583+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:50.163370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":290,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293119+0000","last_change":"2024-03-24T11:26:35.783770+0000","last_active":"2024-03-24T11:27:57.293119+0000","last_peered":"2024-03-24T11:27:57.293119+0000","last_clean":"2024-03-24T11:27:57.293119+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:27:57.293119+0000","last_undegraded":"2024-03-24T11:27:57.293119+0000","last_fullsized":"2024-03-24T11:27:57.293119+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:35.783364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:35.783364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:00.055405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":278,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.316242+0000","last_change":"2024-03-24T11:26:42.302143+0000","last_active":"2024-03-24T11:27:57.316242+0000","last_peered":"2024-03-24T11:27:57.316242+0000","last_clean":"2024-03-24T11:27:57.316242+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:27:57.316242+0000","last_undegraded":"2024-03-24T11:27:57.316242+0000","last_fullsized":"2024-03-24T11:27:57.316242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.301950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:01.472305+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292397+0000","last_change":"2024-03-24T11:26:27.880943+0000","last_active":"2024-03-24T11:27:57.292397+0000","last_peered":"2024-03-24T11:27:57.292397+0000","last_clean":"2024-03-24T11:27:57.292397+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:27:57.292397+0000","last_undegraded":"2024-03-24T11:27:57.292397+0000","last_fullsized":"2024-03-24T11:27:57.292397+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:27.880817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.880817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":318,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296188+0000","last_change":"2024-03-24T11:27:38.277566+0000","last_active":"2024-03-24T11:27:57.296188+0000","last_peered":"2024-03-24T11:27:57.296188+0000","last_clean":"2024-03-24T11:27:57.296188+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:27:57.296188+0000","last_undegraded":"2024-03-24T11:27:57.296188+0000","last_fullsized":"2024-03-24T11:27:57.296188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:27:38.277150+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:27:38.277150+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.877816+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293101+0000","last_change":"2024-03-24T11:27:31.087000+0000","last_active":"2024-03-24T11:27:57.293101+0000","last_peered":"2024-03-24T11:27:57.293101+0000","last_clean":"2024-03-24T11:27:57.293101+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:27:57.293101+0000","last_undegraded":"2024-03-24T11:27:57.293101+0000","last_fullsized":"2024-03-24T11:27:57.293101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:31.086873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:31.086873+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.086873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:58.266470+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":286,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293907+0000","last_change":"2024-03-24T11:25:26.007618+0000","last_active":"2024-03-24T11:27:57.293907+0000","last_peered":"2024-03-24T11:27:57.293907+0000","last_clean":"2024-03-24T11:27:57.293907+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:27:57.293907+0000","last_undegraded":"2024-03-24T11:27:57.293907+0000","last_fullsized":"2024-03-24T11:27:57.293907+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:26.007503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:26.007503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":302,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286570+0000","last_change":"2024-03-24T11:26:30.706204+0000","last_active":"2024-03-24T11:27:57.286570+0000","last_peered":"2024-03-24T11:27:57.286570+0000","last_clean":"2024-03-24T11:27:57.286570+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:27:57.286570+0000","last_undegraded":"2024-03-24T11:27:57.286570+0000","last_fullsized":"2024-03-24T11:27:57.286570+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:30.706071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:30.706071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.520708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"186'9","reported_seq":294,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.316317+0000","last_change":"2024-03-24T11:27:33.522192+0000","last_active":"2024-03-24T11:27:57.316317+0000","last_peered":"2024-03-24T11:27:57.316317+0000","last_clean":"2024-03-24T11:27:57.316317+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:27:57.316317+0000","last_undegraded":"2024-03-24T11:27:57.316317+0000","last_fullsized":"2024-03-24T11:27:57.316317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:27:33.522057+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:55.458927+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"190'13350","reported_seq":13499,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.207600+0000","last_change":"2024-03-24T11:27:27.279069+0000","last_active":"2024-03-24T11:27:58.207600+0000","last_peered":"2024-03-24T11:27:58.207600+0000","last_clean":"2024-03-24T11:27:58.207600+0000","last_became_active":"2024-03-24T11:25:02.676873+0000","last_became_peered":"2024-03-24T11:25:02.676873+0000","last_unstale":"2024-03-24T11:27:58.207600+0000","last_undegraded":"2024-03-24T11:27:58.207600+0000","last_fullsized":"2024-03-24T11:27:58.207600+0000","mapping_epoch":150,"log_start":"178'8300","ondisk_log_start":"178'8300","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'11080","last_scrub_stamp":"2024-03-24T11:27:27.278930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.278930+0000","objects_scrubbed":5005,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:47.009881+0000","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176029696,"num_objects":2686,"num_object_clones":0,"num_object_copies":8058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13350,"num_write_kb":513152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":291,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294017+0000","last_change":"2024-03-24T11:27:36.870198+0000","last_active":"2024-03-24T11:27:57.294017+0000","last_peered":"2024-03-24T11:27:57.294017+0000","last_clean":"2024-03-24T11:27:57.294017+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:27:57.294017+0000","last_undegraded":"2024-03-24T11:27:57.294017+0000","last_fullsized":"2024-03-24T11:27:57.294017+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:36.870117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:36.870117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:58.238790+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":309,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293384+0000","last_change":"2024-03-24T11:26:34.845593+0000","last_active":"2024-03-24T11:27:57.293384+0000","last_peered":"2024-03-24T11:27:57.293384+0000","last_clean":"2024-03-24T11:27:57.293384+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:27:57.293384+0000","last_undegraded":"2024-03-24T11:27:57.293384+0000","last_fullsized":"2024-03-24T11:27:57.293384+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:26:34.845460+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.845460+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:00.943497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"129'22","reported_seq":331,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294027+0000","last_change":"2024-03-24T11:27:37.844432+0000","last_active":"2024-03-24T11:27:57.294027+0000","last_peered":"2024-03-24T11:27:57.294027+0000","last_clean":"2024-03-24T11:27:57.294027+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:27:57.294027+0000","last_undegraded":"2024-03-24T11:27:57.294027+0000","last_fullsized":"2024-03-24T11:27:57.294027+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'22","last_scrub_stamp":"2024-03-24T11:27:37.844238+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:27:37.844238+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.737653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":404,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292985+0000","last_change":"2024-03-24T11:26:48.966305+0000","last_active":"2024-03-24T11:27:57.292985+0000","last_peered":"2024-03-24T11:27:57.292985+0000","last_clean":"2024-03-24T11:27:57.292985+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:27:57.292985+0000","last_undegraded":"2024-03-24T11:27:57.292985+0000","last_fullsized":"2024-03-24T11:27:57.292985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:48.966187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.966187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.360272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"190'13366","reported_seq":13544,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.225228+0000","last_change":"2024-03-24T11:26:46.326986+0000","last_active":"2024-03-24T11:27:58.225228+0000","last_peered":"2024-03-24T11:27:58.225228+0000","last_clean":"2024-03-24T11:27:58.225228+0000","last_became_active":"2024-03-24T11:25:02.675760+0000","last_became_peered":"2024-03-24T11:25:02.675760+0000","last_unstale":"2024-03-24T11:27:58.225228+0000","last_undegraded":"2024-03-24T11:27:58.225228+0000","last_fullsized":"2024-03-24T11:27:58.225228+0000","mapping_epoch":150,"log_start":"176'7500","ondisk_log_start":"176'7500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'8169","last_scrub_stamp":"2024-03-24T11:26:46.326655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.326655+0000","objects_scrubbed":7921,"log_size":5866,"log_dups_size":0,"ondisk_log_size":5866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.124923+0000","scrub_duration":1769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175112192,"num_objects":2672,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13366,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":275,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311868+0000","last_change":"2024-03-24T11:27:26.540096+0000","last_active":"2024-03-24T11:27:57.311868+0000","last_peered":"2024-03-24T11:27:57.311868+0000","last_clean":"2024-03-24T11:27:57.311868+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:27:57.311868+0000","last_undegraded":"2024-03-24T11:27:57.311868+0000","last_fullsized":"2024-03-24T11:27:57.311868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:26.539694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:26.539694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.570356+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293393+0000","last_change":"2024-03-24T11:25:36.259283+0000","last_active":"2024-03-24T11:27:57.293393+0000","last_peered":"2024-03-24T11:27:57.293393+0000","last_clean":"2024-03-24T11:27:57.293393+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:27:57.293393+0000","last_undegraded":"2024-03-24T11:27:57.293393+0000","last_fullsized":"2024-03-24T11:27:57.293393+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:36.259040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:25:36.259040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":314,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284328+0000","last_change":"2024-03-24T11:26:36.876595+0000","last_active":"2024-03-24T11:27:57.284328+0000","last_peered":"2024-03-24T11:27:57.284328+0000","last_clean":"2024-03-24T11:27:57.284328+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:27:57.284328+0000","last_undegraded":"2024-03-24T11:27:57.284328+0000","last_fullsized":"2024-03-24T11:27:57.284328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:26:36.876465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.876465+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120437+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292648+0000","last_change":"2024-03-24T11:26:37.862505+0000","last_active":"2024-03-24T11:27:57.292648+0000","last_peered":"2024-03-24T11:27:57.292648+0000","last_clean":"2024-03-24T11:27:57.292648+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:27:57.292648+0000","last_undegraded":"2024-03-24T11:27:57.292648+0000","last_fullsized":"2024-03-24T11:27:57.292648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.862389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.862389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.892368+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"190'13518","reported_seq":13672,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:59.032399+0000","last_change":"2024-03-24T11:27:18.597916+0000","last_active":"2024-03-24T11:27:59.032399+0000","last_peered":"2024-03-24T11:27:59.032399+0000","last_clean":"2024-03-24T11:27:59.032399+0000","last_became_active":"2024-03-24T11:25:02.676144+0000","last_became_peered":"2024-03-24T11:25:02.676144+0000","last_unstale":"2024-03-24T11:27:59.032399+0000","last_undegraded":"2024-03-24T11:27:59.032399+0000","last_fullsized":"2024-03-24T11:27:59.032399+0000","mapping_epoch":150,"log_start":"176'8000","ondisk_log_start":"176'8000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10592","last_scrub_stamp":"2024-03-24T11:27:18.597618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.597618+0000","objects_scrubbed":5591,"log_size":5518,"log_dups_size":0,"ondisk_log_size":5518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.670378+0000","scrub_duration":3172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":514112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":275,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292155+0000","last_change":"2024-03-24T11:25:33.186360+0000","last_active":"2024-03-24T11:27:57.292155+0000","last_peered":"2024-03-24T11:27:57.292155+0000","last_clean":"2024-03-24T11:27:57.292155+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:27:57.292155+0000","last_undegraded":"2024-03-24T11:27:57.292155+0000","last_fullsized":"2024-03-24T11:27:57.292155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:33.186253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:33.186253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293572+0000","last_change":"2024-03-24T11:26:41.913283+0000","last_active":"2024-03-24T11:27:57.293572+0000","last_peered":"2024-03-24T11:27:57.293572+0000","last_clean":"2024-03-24T11:27:57.293572+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:27:57.293572+0000","last_undegraded":"2024-03-24T11:27:57.293572+0000","last_fullsized":"2024-03-24T11:27:57.293572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.913165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.913165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.067615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293458+0000","last_change":"2024-03-24T11:26:44.921628+0000","last_active":"2024-03-24T11:27:57.293458+0000","last_peered":"2024-03-24T11:27:57.293458+0000","last_clean":"2024-03-24T11:27:57.293458+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:27:57.293458+0000","last_undegraded":"2024-03-24T11:27:57.293458+0000","last_fullsized":"2024-03-24T11:27:57.293458+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:44.921512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:44.921512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:00.943454+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"88'28","reported_seq":339,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286995+0000","last_change":"2024-03-24T11:26:34.694013+0000","last_active":"2024-03-24T11:27:57.286995+0000","last_peered":"2024-03-24T11:27:57.286995+0000","last_clean":"2024-03-24T11:27:57.286995+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:27:57.286995+0000","last_undegraded":"2024-03-24T11:27:57.286995+0000","last_fullsized":"2024-03-24T11:27:57.286995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'28","last_scrub_stamp":"2024-03-24T11:26:34.693862+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.693862+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:01.488304+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"190'13339","reported_seq":13517,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.526998+0000","last_change":"2024-03-24T11:26:44.358566+0000","last_active":"2024-03-24T11:27:58.526998+0000","last_peered":"2024-03-24T11:27:58.526998+0000","last_clean":"2024-03-24T11:27:58.526998+0000","last_became_active":"2024-03-24T11:25:02.599484+0000","last_became_peered":"2024-03-24T11:25:02.599484+0000","last_unstale":"2024-03-24T11:27:58.526998+0000","last_undegraded":"2024-03-24T11:27:58.526998+0000","last_fullsized":"2024-03-24T11:27:58.526998+0000","mapping_epoch":150,"log_start":"179'8500","ondisk_log_start":"179'8500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7990","last_scrub_stamp":"2024-03-24T11:26:44.358125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:44.358125+0000","objects_scrubbed":7882,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:08.519779+0000","scrub_duration":3569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171245568,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13339,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":283,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.298783+0000","last_change":"2024-03-24T11:26:26.644711+0000","last_active":"2024-03-24T11:27:57.298783+0000","last_peered":"2024-03-24T11:27:57.298783+0000","last_clean":"2024-03-24T11:27:57.298783+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:27:57.298783+0000","last_undegraded":"2024-03-24T11:27:57.298783+0000","last_fullsized":"2024-03-24T11:27:57.298783+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.644597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.644597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.270307+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293166+0000","last_change":"2024-03-24T11:26:26.849730+0000","last_active":"2024-03-24T11:27:57.293166+0000","last_peered":"2024-03-24T11:27:57.293166+0000","last_clean":"2024-03-24T11:27:57.293166+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:27:57.293166+0000","last_undegraded":"2024-03-24T11:27:57.293166+0000","last_fullsized":"2024-03-24T11:27:57.293166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.849557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.849557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.064329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":301,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294515+0000","last_change":"2024-03-24T11:25:33.078676+0000","last_active":"2024-03-24T11:27:57.294515+0000","last_peered":"2024-03-24T11:27:57.294515+0000","last_clean":"2024-03-24T11:27:57.294515+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:27:57.294515+0000","last_undegraded":"2024-03-24T11:27:57.294515+0000","last_fullsized":"2024-03-24T11:27:57.294515+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:25:33.078550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:25:33.078550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.841200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":307,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296416+0000","last_change":"2024-03-24T11:26:29.507764+0000","last_active":"2024-03-24T11:27:57.296416+0000","last_peered":"2024-03-24T11:27:57.296416+0000","last_clean":"2024-03-24T11:27:57.296416+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:27:57.296416+0000","last_undegraded":"2024-03-24T11:27:57.296416+0000","last_fullsized":"2024-03-24T11:27:57.296416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:26:29.507604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.507604+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.270219+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"190'13641","reported_seq":13802,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.192450+0000","last_change":"2024-03-24T11:27:15.950757+0000","last_active":"2024-03-24T11:27:58.192450+0000","last_peered":"2024-03-24T11:27:58.192450+0000","last_clean":"2024-03-24T11:27:58.192450+0000","last_became_active":"2024-03-24T11:25:02.675496+0000","last_became_peered":"2024-03-24T11:25:02.675496+0000","last_unstale":"2024-03-24T11:27:58.192450+0000","last_undegraded":"2024-03-24T11:27:58.192450+0000","last_fullsized":"2024-03-24T11:27:58.192450+0000","mapping_epoch":150,"log_start":"179'8600","ondisk_log_start":"179'8600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10572","last_scrub_stamp":"2024-03-24T11:27:15.950518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:15.950518+0000","objects_scrubbed":5747,"log_size":5041,"log_dups_size":0,"ondisk_log_size":5041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.068187+0000","scrub_duration":1924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170983424,"num_objects":2609,"num_object_clones":0,"num_object_copies":7827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13641,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":425,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295208+0000","last_change":"2024-03-24T11:25:48.067965+0000","last_active":"2024-03-24T11:27:57.295208+0000","last_peered":"2024-03-24T11:27:57.295208+0000","last_clean":"2024-03-24T11:27:57.295208+0000","last_became_active":"2024-03-24T11:24:29.093656+0000","last_became_peered":"2024-03-24T11:24:29.093656+0000","last_unstale":"2024-03-24T11:27:57.295208+0000","last_undegraded":"2024-03-24T11:27:57.295208+0000","last_fullsized":"2024-03-24T11:27:57.295208+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:48.067828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:48.067828+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790297+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":275,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.313149+0000","last_change":"2024-03-24T11:26:40.231258+0000","last_active":"2024-03-24T11:27:57.313149+0000","last_peered":"2024-03-24T11:27:57.313149+0000","last_clean":"2024-03-24T11:27:57.313149+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:27:57.313149+0000","last_undegraded":"2024-03-24T11:27:57.313149+0000","last_fullsized":"2024-03-24T11:27:57.313149+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:40.231135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:40.231135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448279+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":306,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293861+0000","last_change":"2024-03-24T11:25:35.099894+0000","last_active":"2024-03-24T11:27:57.293861+0000","last_peered":"2024-03-24T11:27:57.293861+0000","last_clean":"2024-03-24T11:27:57.293861+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:27:57.293861+0000","last_undegraded":"2024-03-24T11:27:57.293861+0000","last_fullsized":"2024-03-24T11:27:57.293861+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:25:35.099746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.099746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.285066+0000","last_change":"2024-03-24T11:26:31.873008+0000","last_active":"2024-03-24T11:27:57.285066+0000","last_peered":"2024-03-24T11:27:57.285066+0000","last_clean":"2024-03-24T11:27:57.285066+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:27:57.285066+0000","last_undegraded":"2024-03-24T11:27:57.285066+0000","last_fullsized":"2024-03-24T11:27:57.285066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:31.872524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:31.872524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":456,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.289014+0000","last_change":"2024-03-24T11:27:13.958327+0000","last_active":"2024-03-24T11:27:57.289014+0000","last_peered":"2024-03-24T11:27:57.289014+0000","last_clean":"2024-03-24T11:27:57.289014+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:27:57.289014+0000","last_undegraded":"2024-03-24T11:27:57.289014+0000","last_fullsized":"2024-03-24T11:27:57.289014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:27:13.958181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:13.958181+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.681771+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296604+0000","last_change":"2024-03-24T11:26:38.524661+0000","last_active":"2024-03-24T11:27:57.296604+0000","last_peered":"2024-03-24T11:27:57.296604+0000","last_clean":"2024-03-24T11:27:57.296604+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:27:57.296604+0000","last_undegraded":"2024-03-24T11:27:57.296604+0000","last_fullsized":"2024-03-24T11:27:57.296604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:38.524541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:38.524541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.968194+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"190'13759","reported_seq":13856,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.714261+0000","last_change":"2024-03-24T11:27:43.581941+0000","last_active":"2024-03-24T11:27:58.714261+0000","last_peered":"2024-03-24T11:27:58.714261+0000","last_clean":"2024-03-24T11:27:58.714261+0000","last_became_active":"2024-03-24T11:25:02.600777+0000","last_became_peered":"2024-03-24T11:25:02.600777+0000","last_unstale":"2024-03-24T11:27:58.714261+0000","last_undegraded":"2024-03-24T11:27:58.714261+0000","last_fullsized":"2024-03-24T11:27:58.714261+0000","mapping_epoch":150,"log_start":"179'9200","ondisk_log_start":"179'9200","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":299,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.489036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176619520,"num_objects":2695,"num_object_clones":0,"num_object_copies":8085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13759,"num_write_kb":526528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":573,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292516+0000","last_change":"2024-03-24T11:27:40.000582+0000","last_active":"2024-03-24T11:27:57.292516+0000","last_peered":"2024-03-24T11:27:57.292516+0000","last_clean":"2024-03-24T11:27:57.292516+0000","last_became_active":"2024-03-24T11:24:29.092038+0000","last_became_peered":"2024-03-24T11:24:29.092038+0000","last_unstale":"2024-03-24T11:27:57.292516+0000","last_undegraded":"2024-03-24T11:27:57.292516+0000","last_fullsized":"2024-03-24T11:27:57.292516+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:27:40.000450+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.673115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":291,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293509+0000","last_change":"2024-03-24T11:27:41.933217+0000","last_active":"2024-03-24T11:27:57.293509+0000","last_peered":"2024-03-24T11:27:57.293509+0000","last_clean":"2024-03-24T11:27:57.293509+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:27:57.293509+0000","last_undegraded":"2024-03-24T11:27:57.293509+0000","last_fullsized":"2024-03-24T11:27:57.293509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:41.933084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:27:41.933084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:56.249244+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":311,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.288828+0000","last_change":"2024-03-24T11:27:34.042910+0000","last_active":"2024-03-24T11:27:57.288828+0000","last_peered":"2024-03-24T11:27:57.288828+0000","last_clean":"2024-03-24T11:27:57.288828+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:27:57.288828+0000","last_undegraded":"2024-03-24T11:27:57.288828+0000","last_fullsized":"2024-03-24T11:27:57.288828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:34.042779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:34.042779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:48.561916+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":312,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294267+0000","last_change":"2024-03-24T11:27:21.956869+0000","last_active":"2024-03-24T11:27:57.294267+0000","last_peered":"2024-03-24T11:27:57.294267+0000","last_clean":"2024-03-24T11:27:57.294267+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:27:57.294267+0000","last_undegraded":"2024-03-24T11:27:57.294267+0000","last_fullsized":"2024-03-24T11:27:57.294267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:21.956404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:27:21.956404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:37.958040+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":301,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294464+0000","last_change":"2024-03-24T11:27:23.937410+0000","last_active":"2024-03-24T11:27:57.294464+0000","last_peered":"2024-03-24T11:27:57.294464+0000","last_clean":"2024-03-24T11:27:57.294464+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:27:57.294464+0000","last_undegraded":"2024-03-24T11:27:57.294464+0000","last_fullsized":"2024-03-24T11:27:57.294464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:27:23.936937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.936937+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:43.630650+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293384+0000","last_change":"2024-03-24T11:26:29.820470+0000","last_active":"2024-03-24T11:27:57.293384+0000","last_peered":"2024-03-24T11:27:57.293384+0000","last_clean":"2024-03-24T11:27:57.293384+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:27:57.293384+0000","last_undegraded":"2024-03-24T11:27:57.293384+0000","last_fullsized":"2024-03-24T11:27:57.293384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:29.820240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.820240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.064292+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"190'13132","reported_seq":13309,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.210888+0000","last_change":"2024-03-24T11:26:48.313995+0000","last_active":"2024-03-24T11:27:58.210888+0000","last_peered":"2024-03-24T11:27:58.210888+0000","last_clean":"2024-03-24T11:27:58.210888+0000","last_became_active":"2024-03-24T11:25:02.600137+0000","last_became_peered":"2024-03-24T11:25:02.600137+0000","last_unstale":"2024-03-24T11:27:58.210888+0000","last_undegraded":"2024-03-24T11:27:58.210888+0000","last_fullsized":"2024-03-24T11:27:58.210888+0000","mapping_epoch":150,"log_start":"176'7300","ondisk_log_start":"176'7300","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8126","last_scrub_stamp":"2024-03-24T11:26:48.313801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.313801+0000","objects_scrubbed":7677,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.465833+0000","scrub_duration":2728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169476096,"num_objects":2586,"num_object_clones":0,"num_object_copies":7758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":291,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293244+0000","last_change":"2024-03-24T11:26:49.074595+0000","last_active":"2024-03-24T11:27:57.293244+0000","last_peered":"2024-03-24T11:27:57.293244+0000","last_clean":"2024-03-24T11:27:57.293244+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:27:57.293244+0000","last_undegraded":"2024-03-24T11:27:57.293244+0000","last_fullsized":"2024-03-24T11:27:57.293244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.074485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:10.184136+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":558,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292360+0000","last_change":"2024-03-24T11:26:36.872094+0000","last_active":"2024-03-24T11:27:57.292360+0000","last_peered":"2024-03-24T11:27:57.292360+0000","last_clean":"2024-03-24T11:27:57.292360+0000","last_became_active":"2024-03-24T11:24:23.650976+0000","last_became_peered":"2024-03-24T11:24:23.650976+0000","last_unstale":"2024-03-24T11:27:57.292360+0000","last_undegraded":"2024-03-24T11:27:57.292360+0000","last_fullsized":"2024-03-24T11:27:57.292360+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:36.871960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.871960+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:00.507528+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":293,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311015+0000","last_change":"2024-03-24T11:27:12.329869+0000","last_active":"2024-03-24T11:27:57.311015+0000","last_peered":"2024-03-24T11:27:57.311015+0000","last_clean":"2024-03-24T11:27:57.311015+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:27:57.311015+0000","last_undegraded":"2024-03-24T11:27:57.311015+0000","last_fullsized":"2024-03-24T11:27:57.311015+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:35.227538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:35.227538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.448182+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"176'24","reported_seq":438,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.287243+0000","last_change":"2024-03-24T11:26:37.676875+0000","last_active":"2024-03-24T11:27:57.287243+0000","last_peered":"2024-03-24T11:27:57.287243+0000","last_clean":"2024-03-24T11:27:57.287243+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:27:57.287243+0000","last_undegraded":"2024-03-24T11:27:57.287243+0000","last_fullsized":"2024-03-24T11:27:57.287243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'24","last_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.676764+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:01.488273+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293532+0000","last_change":"2024-03-24T11:27:11.941555+0000","last_active":"2024-03-24T11:27:57.293532+0000","last_peered":"2024-03-24T11:27:57.293532+0000","last_clean":"2024-03-24T11:27:57.293532+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:27:57.293532+0000","last_undegraded":"2024-03-24T11:27:57.293532+0000","last_fullsized":"2024-03-24T11:27:57.293532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:11.941414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.941414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.864708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"190'13199","reported_seq":13376,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.531687+0000","last_change":"2024-03-24T11:26:42.718381+0000","last_active":"2024-03-24T11:27:58.531687+0000","last_peered":"2024-03-24T11:27:58.531687+0000","last_clean":"2024-03-24T11:27:58.531687+0000","last_became_active":"2024-03-24T11:25:02.674959+0000","last_became_peered":"2024-03-24T11:25:02.674959+0000","last_unstale":"2024-03-24T11:27:58.531687+0000","last_undegraded":"2024-03-24T11:27:58.531687+0000","last_fullsized":"2024-03-24T11:27:58.531687+0000","mapping_epoch":150,"log_start":"178'8300","ondisk_log_start":"178'8300","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7858","last_scrub_stamp":"2024-03-24T11:26:42.718210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.718210+0000","objects_scrubbed":7774,"log_size":4899,"log_dups_size":0,"ondisk_log_size":4899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.515083+0000","scrub_duration":2944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174653440,"num_objects":2665,"num_object_clones":0,"num_object_copies":7995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13199,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":433,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295948+0000","last_change":"2024-03-24T11:27:31.970133+0000","last_active":"2024-03-24T11:27:57.295948+0000","last_peered":"2024-03-24T11:27:57.295948+0000","last_clean":"2024-03-24T11:27:57.295948+0000","last_became_active":"2024-03-24T11:24:29.087723+0000","last_became_peered":"2024-03-24T11:24:29.087723+0000","last_unstale":"2024-03-24T11:27:57.295948+0000","last_undegraded":"2024-03-24T11:27:57.295948+0000","last_fullsized":"2024-03-24T11:27:57.295948+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:31.969994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.969994+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.132766+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":291,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.288564+0000","last_change":"2024-03-24T11:26:42.779061+0000","last_active":"2024-03-24T11:27:57.288564+0000","last_peered":"2024-03-24T11:27:57.288564+0000","last_clean":"2024-03-24T11:27:57.288564+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:27:57.288564+0000","last_undegraded":"2024-03-24T11:27:57.288564+0000","last_fullsized":"2024-03-24T11:27:57.288564+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.778835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.166597+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":298,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.298539+0000","last_change":"2024-03-24T11:26:41.522630+0000","last_active":"2024-03-24T11:27:57.298539+0000","last_peered":"2024-03-24T11:27:57.298539+0000","last_clean":"2024-03-24T11:27:57.298539+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:27:57.298539+0000","last_undegraded":"2024-03-24T11:27:57.298539+0000","last_fullsized":"2024-03-24T11:27:57.298539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.522032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.270279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":844,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292263+0000","last_change":"2024-03-24T11:27:27.024036+0000","last_active":"2024-03-24T11:27:57.292263+0000","last_peered":"2024-03-24T11:27:57.292263+0000","last_clean":"2024-03-24T11:27:57.292263+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:27:57.292263+0000","last_undegraded":"2024-03-24T11:27:57.292263+0000","last_fullsized":"2024-03-24T11:27:57.292263+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:27:27.023866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.023866+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:56.078041+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":537,"num_read_kb":532,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":309,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295894+0000","last_change":"2024-03-24T11:27:11.451262+0000","last_active":"2024-03-24T11:27:57.295894+0000","last_peered":"2024-03-24T11:27:57.295894+0000","last_clean":"2024-03-24T11:27:57.295894+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:27:57.295894+0000","last_undegraded":"2024-03-24T11:27:57.295894+0000","last_fullsized":"2024-03-24T11:27:57.295894+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:27:11.451128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.451128+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.371691+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293830+0000","last_change":"2024-03-24T11:27:17.037155+0000","last_active":"2024-03-24T11:27:57.293830+0000","last_peered":"2024-03-24T11:27:57.293830+0000","last_clean":"2024-03-24T11:27:57.293830+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:27:57.293830+0000","last_undegraded":"2024-03-24T11:27:57.293830+0000","last_fullsized":"2024-03-24T11:27:57.293830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.037043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.037043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.898329+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"190'13451","reported_seq":13542,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.701647+0000","last_change":"2024-03-24T11:25:02.677146+0000","last_active":"2024-03-24T11:27:58.701647+0000","last_peered":"2024-03-24T11:27:58.701647+0000","last_clean":"2024-03-24T11:27:58.701647+0000","last_became_active":"2024-03-24T11:25:02.676840+0000","last_became_peered":"2024-03-24T11:25:02.676840+0000","last_unstale":"2024-03-24T11:27:58.701647+0000","last_undegraded":"2024-03-24T11:27:58.701647+0000","last_fullsized":"2024-03-24T11:27:58.701647+0000","mapping_epoch":150,"log_start":"179'8900","ondisk_log_start":"179'8900","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:03.504409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174391296,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13451,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":431,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.287250+0000","last_change":"2024-03-24T11:26:43.595952+0000","last_active":"2024-03-24T11:27:57.287250+0000","last_peered":"2024-03-24T11:27:57.287250+0000","last_clean":"2024-03-24T11:27:57.287250+0000","last_became_active":"2024-03-24T11:24:29.081667+0000","last_became_peered":"2024-03-24T11:24:29.081667+0000","last_unstale":"2024-03-24T11:27:57.287250+0000","last_undegraded":"2024-03-24T11:27:57.287250+0000","last_fullsized":"2024-03-24T11:27:57.287250+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:43.595832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.595832+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.010557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":291,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295095+0000","last_change":"2024-03-24T11:27:21.158104+0000","last_active":"2024-03-24T11:27:57.295095+0000","last_peered":"2024-03-24T11:27:57.295095+0000","last_clean":"2024-03-24T11:27:57.295095+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:27:57.295095+0000","last_undegraded":"2024-03-24T11:27:57.295095+0000","last_fullsized":"2024-03-24T11:27:57.295095+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:21.157980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:21.157980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:28.941212+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":311,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294147+0000","last_change":"2024-03-24T11:27:25.998316+0000","last_active":"2024-03-24T11:27:57.294147+0000","last_peered":"2024-03-24T11:27:57.294147+0000","last_clean":"2024-03-24T11:27:57.294147+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:27:57.294147+0000","last_undegraded":"2024-03-24T11:27:57.294147+0000","last_fullsized":"2024-03-24T11:27:57.294147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:25.998178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:25.998178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:29.748172+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":327,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296201+0000","last_change":"2024-03-24T11:27:25.297250+0000","last_active":"2024-03-24T11:27:57.296201+0000","last_peered":"2024-03-24T11:27:57.296201+0000","last_clean":"2024-03-24T11:27:57.296201+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:27:57.296201+0000","last_undegraded":"2024-03-24T11:27:57.296201+0000","last_fullsized":"2024-03-24T11:27:57.296201+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-24T11:27:25.297109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:25.297109+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:35.456116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":303,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.287114+0000","last_change":"2024-03-24T11:27:37.391417+0000","last_active":"2024-03-24T11:27:57.287114+0000","last_peered":"2024-03-24T11:27:57.287114+0000","last_clean":"2024-03-24T11:27:57.287114+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:27:57.287114+0000","last_undegraded":"2024-03-24T11:27:57.287114+0000","last_fullsized":"2024-03-24T11:27:57.287114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:37.391291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:38.330738+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"190'13411","reported_seq":13565,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.987086+0000","last_change":"2024-03-24T11:27:20.326173+0000","last_active":"2024-03-24T11:27:58.987086+0000","last_peered":"2024-03-24T11:27:58.987086+0000","last_clean":"2024-03-24T11:27:58.987086+0000","last_became_active":"2024-03-24T11:25:02.676938+0000","last_became_peered":"2024-03-24T11:25:02.676938+0000","last_unstale":"2024-03-24T11:27:58.987086+0000","last_undegraded":"2024-03-24T11:27:58.987086+0000","last_fullsized":"2024-03-24T11:27:58.987086+0000","mapping_epoch":150,"log_start":"176'7900","ondisk_log_start":"176'7900","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10606","last_scrub_stamp":"2024-03-24T11:27:20.325901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:20.325901+0000","objects_scrubbed":5503,"log_size":5511,"log_dups_size":0,"ondisk_log_size":5511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:46.812470+0000","scrub_duration":2923,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170196992,"num_objects":2597,"num_object_clones":0,"num_object_copies":7791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13411,"num_write_kb":512256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":431,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.287433+0000","last_change":"2024-03-24T11:26:32.673765+0000","last_active":"2024-03-24T11:27:57.287433+0000","last_peered":"2024-03-24T11:27:57.287433+0000","last_clean":"2024-03-24T11:27:57.287433+0000","last_became_active":"2024-03-24T11:24:29.117106+0000","last_became_peered":"2024-03-24T11:24:29.117106+0000","last_unstale":"2024-03-24T11:27:57.287433+0000","last_undegraded":"2024-03-24T11:27:57.287433+0000","last_fullsized":"2024-03-24T11:27:57.287433+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:32.673525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.673525+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.520752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":284,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295618+0000","last_change":"2024-03-24T11:27:27.391992+0000","last_active":"2024-03-24T11:27:57.295618+0000","last_peered":"2024-03-24T11:27:57.295618+0000","last_clean":"2024-03-24T11:27:57.295618+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:27:57.295618+0000","last_undegraded":"2024-03-24T11:27:57.295618+0000","last_fullsized":"2024-03-24T11:27:57.295618+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:27:27.391857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.391857+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:33.529518+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":306,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293026+0000","last_change":"2024-03-24T11:27:27.949051+0000","last_active":"2024-03-24T11:27:57.293026+0000","last_peered":"2024-03-24T11:27:57.293026+0000","last_clean":"2024-03-24T11:27:57.293026+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:27:57.293026+0000","last_undegraded":"2024-03-24T11:27:57.293026+0000","last_fullsized":"2024-03-24T11:27:57.293026+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:27.948926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.948926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.013446+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":314,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293878+0000","last_change":"2024-03-24T11:26:49.790900+0000","last_active":"2024-03-24T11:27:57.293878+0000","last_peered":"2024-03-24T11:27:57.293878+0000","last_clean":"2024-03-24T11:27:57.293878+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:27:57.293878+0000","last_undegraded":"2024-03-24T11:27:57.293878+0000","last_fullsized":"2024-03-24T11:27:57.293878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:26:49.790753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.790753+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:09.066698+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":414,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.296539+0000","last_change":"2024-03-24T11:27:34.364048+0000","last_active":"2024-03-24T11:27:57.296539+0000","last_peered":"2024-03-24T11:27:57.296539+0000","last_clean":"2024-03-24T11:27:57.296539+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:27:57.296539+0000","last_undegraded":"2024-03-24T11:27:57.296539+0000","last_fullsized":"2024-03-24T11:27:57.296539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:27:34.363907+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:34.363907+0000","last_clean_scrub_stamp":"2024-03-24T11:27:34.363907+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:53.265803+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"190'13342","reported_seq":13518,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.213243+0000","last_change":"2024-03-24T11:26:49.441112+0000","last_active":"2024-03-24T11:27:58.213243+0000","last_peered":"2024-03-24T11:27:58.213243+0000","last_clean":"2024-03-24T11:27:58.213243+0000","last_became_active":"2024-03-24T11:25:02.675745+0000","last_became_peered":"2024-03-24T11:25:02.675745+0000","last_unstale":"2024-03-24T11:27:58.213243+0000","last_undegraded":"2024-03-24T11:27:58.213243+0000","last_fullsized":"2024-03-24T11:27:58.213243+0000","mapping_epoch":150,"log_start":"178'8300","ondisk_log_start":"178'8300","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8346","last_scrub_stamp":"2024-03-24T11:26:49.440843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.440843+0000","objects_scrubbed":7686,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.854214+0000","scrub_duration":2353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170459160,"num_objects":2602,"num_object_clones":0,"num_object_copies":7806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13342,"num_write_kb":510145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":425,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311531+0000","last_change":"2024-03-24T11:27:36.545219+0000","last_active":"2024-03-24T11:27:57.311531+0000","last_peered":"2024-03-24T11:27:57.311531+0000","last_clean":"2024-03-24T11:27:57.311531+0000","last_became_active":"2024-03-24T11:24:29.087468+0000","last_became_peered":"2024-03-24T11:24:29.087468+0000","last_unstale":"2024-03-24T11:27:57.311531+0000","last_undegraded":"2024-03-24T11:27:57.311531+0000","last_fullsized":"2024-03-24T11:27:57.311531+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:36.545078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:36.545078+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:37.180498+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":283,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292758+0000","last_change":"2024-03-24T11:27:28.077066+0000","last_active":"2024-03-24T11:27:57.292758+0000","last_peered":"2024-03-24T11:27:57.292758+0000","last_clean":"2024-03-24T11:27:57.292758+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:27:57.292758+0000","last_undegraded":"2024-03-24T11:27:57.292758+0000","last_fullsized":"2024-03-24T11:27:57.292758+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:28.076928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:28.076928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:45.991016+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":304,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295991+0000","last_change":"2024-03-24T11:27:41.223227+0000","last_active":"2024-03-24T11:27:57.295991+0000","last_peered":"2024-03-24T11:27:57.295991+0000","last_clean":"2024-03-24T11:27:57.295991+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:27:57.295991+0000","last_undegraded":"2024-03-24T11:27:57.295991+0000","last_fullsized":"2024-03-24T11:27:57.295991+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:27:41.223105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:41.223105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:56.305173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":312,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284237+0000","last_change":"2024-03-24T11:27:15.813035+0000","last_active":"2024-03-24T11:27:57.284237+0000","last_peered":"2024-03-24T11:27:57.284237+0000","last_clean":"2024-03-24T11:27:57.284237+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:27:57.284237+0000","last_undegraded":"2024-03-24T11:27:57.284237+0000","last_fullsized":"2024-03-24T11:27:57.284237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-03-24T11:27:15.812897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:15.812897+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:19.144490+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":388,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.310751+0000","last_change":"2024-03-24T11:25:20.232658+0000","last_active":"2024-03-24T11:27:57.310751+0000","last_peered":"2024-03-24T11:27:57.310751+0000","last_clean":"2024-03-24T11:27:57.310751+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:27:57.310751+0000","last_undegraded":"2024-03-24T11:27:57.310751+0000","last_fullsized":"2024-03-24T11:27:57.310751+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:20.232540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:20.232540+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.446399+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"190'13208","reported_seq":13364,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.994208+0000","last_change":"2024-03-24T11:27:18.681143+0000","last_active":"2024-03-24T11:27:58.994208+0000","last_peered":"2024-03-24T11:27:58.994208+0000","last_clean":"2024-03-24T11:27:58.994208+0000","last_became_active":"2024-03-24T11:25:02.675706+0000","last_became_peered":"2024-03-24T11:25:02.675706+0000","last_unstale":"2024-03-24T11:27:58.994208+0000","last_undegraded":"2024-03-24T11:27:58.994208+0000","last_fullsized":"2024-03-24T11:27:58.994208+0000","mapping_epoch":150,"log_start":"176'7700","ondisk_log_start":"176'7700","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10233","last_scrub_stamp":"2024-03-24T11:27:18.680945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.680945+0000","objects_scrubbed":5603,"log_size":5508,"log_dups_size":0,"ondisk_log_size":5508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:20.438526+0000","scrub_duration":2239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168427520,"num_objects":2570,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13208,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":425,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.285420+0000","last_change":"2024-03-24T11:26:29.924513+0000","last_active":"2024-03-24T11:27:57.285420+0000","last_peered":"2024-03-24T11:27:57.285420+0000","last_clean":"2024-03-24T11:27:57.285420+0000","last_became_active":"2024-03-24T11:24:29.082786+0000","last_became_peered":"2024-03-24T11:24:29.082786+0000","last_unstale":"2024-03-24T11:27:57.285420+0000","last_undegraded":"2024-03-24T11:27:57.285420+0000","last_fullsized":"2024-03-24T11:27:57.285420+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:29.924076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.924076+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120525+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":283,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284622+0000","last_change":"2024-03-24T11:27:23.794256+0000","last_active":"2024-03-24T11:27:57.284622+0000","last_peered":"2024-03-24T11:27:57.284622+0000","last_clean":"2024-03-24T11:27:57.284622+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:27:57.284622+0000","last_undegraded":"2024-03-24T11:27:57.284622+0000","last_fullsized":"2024-03-24T11:27:57.284622+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:23.794181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:49.118465+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.794181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:49.002523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.316069+0000","last_change":"2024-03-24T11:26:39.266713+0000","last_active":"2024-03-24T11:27:57.316069+0000","last_peered":"2024-03-24T11:27:57.316069+0000","last_clean":"2024-03-24T11:27:57.316069+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:27:57.316069+0000","last_undegraded":"2024-03-24T11:27:57.316069+0000","last_fullsized":"2024-03-24T11:27:57.316069+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.265692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.265692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.928492+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":307,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.310756+0000","last_change":"2024-03-24T11:25:35.339628+0000","last_active":"2024-03-24T11:27:57.310756+0000","last_peered":"2024-03-24T11:27:57.310756+0000","last_clean":"2024-03-24T11:27:57.310756+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:27:57.310756+0000","last_undegraded":"2024-03-24T11:27:57.310756+0000","last_fullsized":"2024-03-24T11:27:57.310756+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-24T11:25:35.339383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.339383+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.447961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.283992+0000","last_change":"2024-03-24T11:26:28.950793+0000","last_active":"2024-03-24T11:27:57.283992+0000","last_peered":"2024-03-24T11:27:57.283992+0000","last_clean":"2024-03-24T11:27:57.283992+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:27:57.283992+0000","last_undegraded":"2024-03-24T11:27:57.283992+0000","last_fullsized":"2024-03-24T11:27:57.283992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.950670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.950670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.120378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"190'13373","reported_seq":13546,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.504059+0000","last_change":"2024-03-24T11:26:34.774580+0000","last_active":"2024-03-24T11:27:58.504059+0000","last_peered":"2024-03-24T11:27:58.504059+0000","last_clean":"2024-03-24T11:27:58.504059+0000","last_became_active":"2024-03-24T11:25:02.600315+0000","last_became_peered":"2024-03-24T11:25:02.600315+0000","last_unstale":"2024-03-24T11:27:58.504059+0000","last_undegraded":"2024-03-24T11:27:58.504059+0000","last_fullsized":"2024-03-24T11:27:58.504059+0000","mapping_epoch":150,"log_start":"179'8500","ondisk_log_start":"179'8500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7439","last_scrub_stamp":"2024-03-24T11:26:34.774083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.774083+0000","objects_scrubbed":7347,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.261755+0000","scrub_duration":2963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177405952,"num_objects":2707,"num_object_clones":0,"num_object_copies":8121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13373,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":425,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293732+0000","last_change":"2024-03-24T11:25:40.019964+0000","last_active":"2024-03-24T11:27:57.293732+0000","last_peered":"2024-03-24T11:27:57.293732+0000","last_clean":"2024-03-24T11:27:57.293732+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:27:57.293732+0000","last_undegraded":"2024-03-24T11:27:57.293732+0000","last_fullsized":"2024-03-24T11:27:57.293732+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:40.019881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.019881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292760+0000","last_change":"2024-03-24T11:27:42.911979+0000","last_active":"2024-03-24T11:27:57.292760+0000","last_peered":"2024-03-24T11:27:57.292760+0000","last_clean":"2024-03-24T11:27:57.292760+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:27:57.292760+0000","last_undegraded":"2024-03-24T11:27:57.292760+0000","last_fullsized":"2024-03-24T11:27:57.292760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:42.911873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:42.911873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:12.605156+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":404,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286240+0000","last_change":"2024-03-24T11:26:42.827014+0000","last_active":"2024-03-24T11:27:57.286240+0000","last_peered":"2024-03-24T11:27:57.286240+0000","last_clean":"2024-03-24T11:27:57.286240+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:27:57.286240+0000","last_undegraded":"2024-03-24T11:27:57.286240+0000","last_fullsized":"2024-03-24T11:27:57.286240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:42.826873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.826873+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:04.232390+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.310609+0000","last_change":"2024-03-24T11:25:22.291763+0000","last_active":"2024-03-24T11:27:57.310609+0000","last_peered":"2024-03-24T11:27:57.310609+0000","last_clean":"2024-03-24T11:27:57.310609+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:27:57.310609+0000","last_undegraded":"2024-03-24T11:27:57.310609+0000","last_fullsized":"2024-03-24T11:27:57.310609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:22.291490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:25:22.291490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.447993+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"89'22","reported_seq":331,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.295355+0000","last_change":"2024-03-24T11:26:46.757168+0000","last_active":"2024-03-24T11:27:57.295355+0000","last_peered":"2024-03-24T11:27:57.295355+0000","last_clean":"2024-03-24T11:27:57.295355+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:27:57.295355+0000","last_undegraded":"2024-03-24T11:27:57.295355+0000","last_fullsized":"2024-03-24T11:27:57.295355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-03-24T11:26:46.757070+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.757070+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.851634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"190'13089","reported_seq":13264,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.139129+0000","last_change":"2024-03-24T11:26:48.243638+0000","last_active":"2024-03-24T11:27:58.139129+0000","last_peered":"2024-03-24T11:27:58.139129+0000","last_clean":"2024-03-24T11:27:58.139129+0000","last_became_active":"2024-03-24T11:25:02.599329+0000","last_became_peered":"2024-03-24T11:25:02.599329+0000","last_unstale":"2024-03-24T11:27:58.139129+0000","last_undegraded":"2024-03-24T11:27:58.139129+0000","last_fullsized":"2024-03-24T11:27:58.139129+0000","mapping_epoch":150,"log_start":"179'8500","ondisk_log_start":"179'8500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8117","last_scrub_stamp":"2024-03-24T11:26:48.243210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.243210+0000","objects_scrubbed":7656,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:02.207893+0000","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13089,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":275,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311675+0000","last_change":"2024-03-24T11:27:17.423048+0000","last_active":"2024-03-24T11:27:57.311675+0000","last_peered":"2024-03-24T11:27:57.311675+0000","last_clean":"2024-03-24T11:27:57.311675+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:27:57.311675+0000","last_undegraded":"2024-03-24T11:27:57.311675+0000","last_fullsized":"2024-03-24T11:27:57.311675+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.422927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.062528+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":286,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293748+0000","last_change":"2024-03-24T11:26:28.105930+0000","last_active":"2024-03-24T11:27:57.293748+0000","last_peered":"2024-03-24T11:27:57.293748+0000","last_clean":"2024-03-24T11:27:57.293748+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:27:57.293748+0000","last_undegraded":"2024-03-24T11:27:57.293748+0000","last_fullsized":"2024-03-24T11:27:57.293748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.105784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:24:43.042611+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.105784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:00.786537+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284075+0000","last_change":"2024-03-24T11:26:49.883140+0000","last_active":"2024-03-24T11:27:57.284075+0000","last_peered":"2024-03-24T11:27:57.284075+0000","last_clean":"2024-03-24T11:27:57.284075+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:27:57.284075+0000","last_undegraded":"2024-03-24T11:27:57.284075+0000","last_fullsized":"2024-03-24T11:27:57.284075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.883061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.883061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.057605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":318,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286063+0000","last_change":"2024-03-24T11:27:26.870931+0000","last_active":"2024-03-24T11:27:57.286063+0000","last_peered":"2024-03-24T11:27:57.286063+0000","last_clean":"2024-03-24T11:27:57.286063+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:27:57.286063+0000","last_undegraded":"2024-03-24T11:27:57.286063+0000","last_fullsized":"2024-03-24T11:27:57.286063+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:27:26.870675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:26.870675+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:53.427130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"190'13381","reported_seq":13556,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.531924+0000","last_change":"2024-03-24T11:26:36.863675+0000","last_active":"2024-03-24T11:27:58.531924+0000","last_peered":"2024-03-24T11:27:58.531924+0000","last_clean":"2024-03-24T11:27:58.531924+0000","last_became_active":"2024-03-24T11:25:02.674932+0000","last_became_peered":"2024-03-24T11:25:02.674932+0000","last_unstale":"2024-03-24T11:27:58.531924+0000","last_undegraded":"2024-03-24T11:27:58.531924+0000","last_fullsized":"2024-03-24T11:27:58.531924+0000","mapping_epoch":150,"log_start":"179'8500","ondisk_log_start":"179'8500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7579","last_scrub_stamp":"2024-03-24T11:26:36.863361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.863361+0000","objects_scrubbed":7488,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:06.565935+0000","scrub_duration":3086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174391296,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13381,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":426,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293737+0000","last_change":"2024-03-24T11:25:43.004840+0000","last_active":"2024-03-24T11:27:57.293737+0000","last_peered":"2024-03-24T11:27:57.293737+0000","last_clean":"2024-03-24T11:27:57.293737+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:27:57.293737+0000","last_undegraded":"2024-03-24T11:27:57.293737+0000","last_fullsized":"2024-03-24T11:27:57.293737+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:25:43.004596+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:25:43.004596+0000","last_clean_scrub_stamp":"2024-03-24T11:25:43.004596+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:57.790055+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":290,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294713+0000","last_change":"2024-03-24T11:27:19.056330+0000","last_active":"2024-03-24T11:27:57.294713+0000","last_peered":"2024-03-24T11:27:57.294713+0000","last_clean":"2024-03-24T11:27:57.294713+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:27:57.294713+0000","last_undegraded":"2024-03-24T11:27:57.294713+0000","last_fullsized":"2024-03-24T11:27:57.294713+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:19.055992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:19.055992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:28.067254+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":295,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.294117+0000","last_change":"2024-03-24T11:26:48.793731+0000","last_active":"2024-03-24T11:27:57.294117+0000","last_peered":"2024-03-24T11:27:57.294117+0000","last_clean":"2024-03-24T11:27:57.294117+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:27:57.294117+0000","last_undegraded":"2024-03-24T11:27:57.294117+0000","last_fullsized":"2024-03-24T11:27:57.294117+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.793614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.316226+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":317,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286920+0000","last_change":"2024-03-24T11:27:22.478172+0000","last_active":"2024-03-24T11:27:57.286920+0000","last_peered":"2024-03-24T11:27:57.286920+0000","last_clean":"2024-03-24T11:27:57.286920+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:27:57.286920+0000","last_undegraded":"2024-03-24T11:27:57.286920+0000","last_fullsized":"2024-03-24T11:27:57.286920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:27:22.477947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:22.477947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.862237+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"88'25","reported_seq":359,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286171+0000","last_change":"2024-03-24T11:27:42.047337+0000","last_active":"2024-03-24T11:27:57.286171+0000","last_peered":"2024-03-24T11:27:57.286171+0000","last_clean":"2024-03-24T11:27:57.286171+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:27:57.286171+0000","last_undegraded":"2024-03-24T11:27:57.286171+0000","last_fullsized":"2024-03-24T11:27:57.286171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'25","last_scrub_stamp":"2024-03-24T11:27:42.047201+0000","last_deep_scrub":"88'25","last_deep_scrub_stamp":"2024-03-24T11:25:49.109688+0000","last_clean_scrub_stamp":"2024-03-24T11:27:42.047201+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:54.568794+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"190'13425","reported_seq":13516,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:58.697989+0000","last_change":"2024-03-24T11:25:02.599491+0000","last_active":"2024-03-24T11:27:58.697989+0000","last_peered":"2024-03-24T11:27:58.697989+0000","last_clean":"2024-03-24T11:27:58.697989+0000","last_became_active":"2024-03-24T11:25:02.599177+0000","last_became_peered":"2024-03-24T11:25:02.599177+0000","last_unstale":"2024-03-24T11:27:58.697989+0000","last_undegraded":"2024-03-24T11:27:58.697989+0000","last_fullsized":"2024-03-24T11:27:58.697989+0000","mapping_epoch":150,"log_start":"179'8900","ondisk_log_start":"179'8900","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4525,"log_dups_size":0,"ondisk_log_size":4525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:03.504391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169934848,"num_objects":2593,"num_object_clones":0,"num_object_copies":7779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13425,"num_write_kb":512576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":573,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.292187+0000","last_change":"2024-03-24T11:27:12.962801+0000","last_active":"2024-03-24T11:27:57.292187+0000","last_peered":"2024-03-24T11:27:57.292187+0000","last_clean":"2024-03-24T11:27:57.292187+0000","last_became_active":"2024-03-24T11:24:29.091858+0000","last_became_peered":"2024-03-24T11:24:29.091858+0000","last_unstale":"2024-03-24T11:27:57.292187+0000","last_undegraded":"2024-03-24T11:27:57.292187+0000","last_fullsized":"2024-03-24T11:27:57.292187+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:12.962703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:12.962703+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.888591+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":282,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.286125+0000","last_change":"2024-03-24T11:26:37.859404+0000","last_active":"2024-03-24T11:27:57.286125+0000","last_peered":"2024-03-24T11:27:57.286125+0000","last_clean":"2024-03-24T11:27:57.286125+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:27:57.286125+0000","last_undegraded":"2024-03-24T11:27:57.286125+0000","last_fullsized":"2024-03-24T11:27:57.286125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.859286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.859286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:59.078541+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":420,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.293284+0000","last_change":"2024-03-24T11:27:19.940085+0000","last_active":"2024-03-24T11:27:57.293284+0000","last_peered":"2024-03-24T11:27:57.293284+0000","last_clean":"2024-03-24T11:27:57.293284+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:27:57.293284+0000","last_undegraded":"2024-03-24T11:27:57.293284+0000","last_fullsized":"2024-03-24T11:27:57.293284+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:27:19.939971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:19.939971+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:47.560194+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":287,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.311559+0000","last_change":"2024-03-24T11:26:46.366056+0000","last_active":"2024-03-24T11:27:57.311559+0000","last_peered":"2024-03-24T11:27:57.311559+0000","last_clean":"2024-03-24T11:27:57.311559+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:27:57.311559+0000","last_undegraded":"2024-03-24T11:27:57.311559+0000","last_fullsized":"2024-03-24T11:27:57.311559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.365877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.365877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:58.222226+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":306,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-03-24T11:27:57.284785+0000","last_change":"2024-03-24T11:27:35.003213+0000","last_active":"2024-03-24T11:27:57.284785+0000","last_peered":"2024-03-24T11:27:57.284785+0000","last_clean":"2024-03-24T11:27:57.284785+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:27:57.284785+0000","last_undegraded":"2024-03-24T11:27:57.284785+0000","last_fullsized":"2024-03-24T11:27:57.284785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:27:35.003081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:35.003081+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:40.552259+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2758737944,"num_objects":42096,"num_object_clones":0,"num_object_copies":126288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213984,"num_write_kb":8194497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8301654016,"data_stored":8301641800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81084,"ondisk_log_size":81084,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":537,"num_read_kb":532,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":451,"ondisk_log_size":451,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070577,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1103848,"kb_used_data":847708,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92641304,"statfs":{"total":95995035648,"available":94864695296,"internally_reserved":0,"allocated":868052992,"data_stored":867729716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":262207947},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364918,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080280,"kb_used_data":864460,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92664872,"statfs":{"total":95995035648,"available":94888828928,"internally_reserved":0,"allocated":885207040,"data_stored":884900345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1471552,"kb_used_data":1205620,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92273600,"statfs":{"total":95995035648,"available":94488166400,"internally_reserved":0,"allocated":1234554880,"data_stored":1234208062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1345004,"kb_used_data":1012512,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":92400148,"statfs":{"total":95995035648,"available":94617751552,"internally_reserved":0,"allocated":1036812288,"data_stored":1036484647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1294440,"kb_used_data":1032796,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92450712,"statfs":{"total":95995035648,"available":94669529088,"internally_reserved":0,"allocated":1057583104,"data_stored":1057259888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1306624,"kb_used_data":1014388,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92438528,"statfs":{"total":95995035648,"available":94657052672,"internally_reserved":0,"allocated":1038733312,"data_stored":1038386230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986304,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1085808,"kb_used_data":860260,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92659344,"statfs":{"total":95995035648,"available":94883168256,"internally_reserved":0,"allocated":880906240,"data_stored":880575176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659259,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1626404,"kb_used_data":1370328,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92118748,"statfs":{"total":95995035648,"available":94329597952,"internally_reserved":0,"allocated":1403215872,"data_stored":1402897018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1012203520,"data_stored":1012203520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1222115328,"data_stored":1222115328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1041829888,"data_stored":1041825816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1028915200,"data_stored":1028915200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":879230976,"data_stored":879230976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1387466752,"data_stored":1387462680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":867237888,"data_stored":867237888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":862654464,"data_stored":862650392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:28:04.819 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-03-24T11:28:04.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:04.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:04.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:04.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:05.013 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:05.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:05.149 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:05.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:05.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:05.337 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:28:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:28:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:28:05.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:05.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:05.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:05.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:05.609 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:05 smithi179 ceph-mon[29608]: pgmap v366: 158 pgs: 158 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:28:05.610 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:05 smithi179 ceph-mon[37907]: pgmap v366: 158 pgs: 158 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:28:05.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:05.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:05 smithi181 ceph-mon[33998]: pgmap v366: 158 pgs: 158 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:28:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.694 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:05.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:05.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:05.826 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:05.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:05.964 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:05.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:06.066 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:06.108 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:06.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:06.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:06.247 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:06.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:06.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:06.389 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:06.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:06.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:06.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:06.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:06.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:06.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:06.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:06.757 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:28:06.757 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:28:06.757 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:28:06.758 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:28:06.758 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:28:06.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:06.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:06.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:06.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:06.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:06.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:07.031 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:07.074 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:07.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:07.175 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:07.226 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:07.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:07.328 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:07.375 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:07.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:07.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:07.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:07.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:07.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:07.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:07.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:07.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:07 smithi181 ceph-mon[33998]: pgmap v367: 158 pgs: 158 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:28:07.680 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:07.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:07 smithi179 ceph-mon[37907]: pgmap v367: 158 pgs: 158 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:28:07.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:07 smithi179 ceph-mon[29608]: pgmap v367: 158 pgs: 158 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-24T11:28:07.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:07.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:07.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:07.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:07.913 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:07.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:08.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:08.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:08.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:08.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:08.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:08.286 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:28:08.314 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:08.392 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:28:08.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:08.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:08.533 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:08 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/222133785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:28:08.533 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:08 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:28:08.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:08 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/222133785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:28:08.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:08 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:28:08.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:08.588 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:08.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:08 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/222133785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:28:08.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:08 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:28:08.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:08.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:08.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:08.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:08.956 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:08.987 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:08.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:09.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:09.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:09.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:09.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:09.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:09.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:09.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:09.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:09.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:09.492 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:09.533 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:09.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:09.634 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:09.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:09 smithi181 ceph-mon[33998]: pgmap v368: 158 pgs: 158 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:28:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:09 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:28:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:09 smithi181 ceph-mon[33998]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:28:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:09 smithi181 ceph-mon[33998]: 1.8 scrub starts 2024-03-24T11:28:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:09 smithi181 ceph-mon[33998]: 1.8 scrub ok 2024-03-24T11:28:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:09.670 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:09.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[29608]: pgmap v368: 158 pgs: 158 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:28:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[29608]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[29608]: 1.8 scrub starts 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[29608]: 1.8 scrub ok 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[37907]: pgmap v368: 158 pgs: 158 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[37907]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[37907]: 1.8 scrub starts 2024-03-24T11:28:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:09 smithi179 ceph-mon[37907]: 1.8 scrub ok 2024-03-24T11:28:09.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:09.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:09.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:09.819 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:28:09.819 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:28:09.820 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-24T11:28:09.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:28:09.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:09.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:09.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:10.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:10.086 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:10.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:10.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:10.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:10.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:10.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:10.381 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:10.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:10.483 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:10.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:10.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:10.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:10.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:10 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:28:10.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:10 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:28:10.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:10 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:28:10.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:10 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:28:10.666 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:10.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:28:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:28:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:28:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:28:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:28:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:28:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:28:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:10 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:28:10.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:10.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:10.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:10.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:10.946 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:10.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:11.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:11.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:11.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:11.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:11.228 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:11.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:11.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:11.368 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:11.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:11.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:11.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:11.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:11.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:11.645 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:11 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:28:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:11 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:28:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:11 smithi181 ceph-mon[33998]: pgmap v370: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:28:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:11 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:28:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:11 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:28:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:11.668 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:11.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:11.685 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[29608]: pgmap v370: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[37907]: pgmap v370: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:28:11.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:11 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:28:11.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:11.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:11.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:11.905 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:11.947 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:11.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:12.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:12.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:12.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:12.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:12.227 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:12.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:12.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:12.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:12.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:12.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:28:12.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:12.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:12.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:12.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:12.640 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:12.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:28:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:28:12.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:28:12.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:28:12.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:12 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:28:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:28:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:28:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:28:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:28:12.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:12 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:28:12.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:12.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:12.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:12.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:12.919 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:12.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:13.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:13.062 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:13.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:13.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:13.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:13.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:13.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:13.332 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:13.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:13.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:28:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:28:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:28:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: pgmap v372: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 8.b scrub starts 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:28:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:28:13.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:28:13.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:28:13.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: pgmap v372: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s 2024-03-24T11:28:13.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 8.b scrub starts 2024-03-24T11:28:13.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:28:13.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:13 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:28:13.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:13.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:13.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:13.629 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:13.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:13.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:13.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:13.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:13.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:13.896 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:13.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:13.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:28:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:28:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:28:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:28:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:28:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:28:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:28:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: pgmap v372: 158 pgs: 158 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s 2024-03-24T11:28:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 8.b scrub starts 2024-03-24T11:28:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:28:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:13 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:28:13.988 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:28:13.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:14.040 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:28:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:14.044 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:14.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:14.080 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:28:14.122 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:28:14.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:14.161 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:28:14.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:14.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:14.202 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:28:14.275 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:14.282 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:28:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:14.317 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:14.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:14.318 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:28:14.334 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:28:14.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:28:14.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:28:14.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:28:14.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:28:14.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:28:14.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:28:14.335 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:28:14.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:14.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:14.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:14.534 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:28:14.535 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:28:14.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:14.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:14.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:14.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:28:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:28:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:28:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[29608]: 8.b scrub ok 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3686575221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[37907]: 8.b scrub ok 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:14 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3686575221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:14.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:14.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:14.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:14.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:14.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:14 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:28:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:14 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:28:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:14 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:28:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:14 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:28:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:14 smithi181 ceph-mon[33998]: 8.b scrub ok 2024-03-24T11:28:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:14 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3686575221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:14.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:15.007 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:15.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:15.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:15.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:15.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:15.308 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:28:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:28:15.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:15.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:15.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:15.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:15.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: pgmap v373: 158 pgs: 158 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 5.0 scrub starts 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 5.0 scrub ok 2024-03-24T11:28:15.584 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[29608]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: pgmap v373: 158 pgs: 158 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 5.0 scrub starts 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 5.0 scrub ok 2024-03-24T11:28:15.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:28:15.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:28:15.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:15 smithi179 ceph-mon[37907]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:28:15.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:15.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:15.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:15.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: pgmap v373: 158 pgs: 158 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 5.0 scrub starts 2024-03-24T11:28:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 5.0 scrub ok 2024-03-24T11:28:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:28:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:28:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:15 smithi181 ceph-mon[33998]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T11:28:15.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:15.985 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:15.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:16.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:16.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:16.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:16.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:16.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:16.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:16.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:16.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:16.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:16.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:16.526 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:16.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:16.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:16.666 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:16.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:16.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:28:16.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:28:16.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:28:16.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:28:16.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:28:16.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:28:16.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:28:16.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:28:16.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:28:16.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:28:16.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:28:16.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:16 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:28:16.702 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:28:16.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:16.799 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:16.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:16.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:16 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:28:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:16 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:28:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:16 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:28:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:16 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:28:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:16 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:28:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:16 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:28:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:16.932 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:16.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:17.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:17.074 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:17.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:17.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:28:17.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:17.243 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:17.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:17.292 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:17.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:17.373 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:17.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:17.446 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[29608]: pgmap v375: 158 pgs: 158 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:28:17.446 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:28:17.446 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:28:17.446 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[29608]: 8.0 scrub starts 2024-03-24T11:28:17.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[37907]: pgmap v375: 158 pgs: 158 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:28:17.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:28:17.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:28:17.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:17 smithi179 ceph-mon[37907]: 8.0 scrub starts 2024-03-24T11:28:17.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:17.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:17.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:17.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:17.639 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:17.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:17.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:17.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:17.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:17.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:17 smithi181 ceph-mon[33998]: pgmap v375: 158 pgs: 158 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:28:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:17 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:28:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:17 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:28:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:17 smithi181 ceph-mon[33998]: 8.0 scrub starts 2024-03-24T11:28:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:17.910 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:17.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:18.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:18.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:18.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:18.150 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:18.187 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:18.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:18.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:18.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:18.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:18.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:18.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:18.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:18.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:18.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:18.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:18.646 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:18.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:18.713 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:28:18.713 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:28:18.713 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[29608]: 8.0 scrub ok 2024-03-24T11:28:18.713 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:18.713 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:28:18.713 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:28:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:28:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:28:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[37907]: 8.0 scrub ok 2024-03-24T11:28:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:28:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:18 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:28:18.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:18.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:18.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:18.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:18.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:18.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:18 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:28:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:18 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:28:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:18 smithi181 ceph-mon[33998]: 8.0 scrub ok 2024-03-24T11:28:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:18 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:28:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:18 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:28:18.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:19.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:19.090 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:28:19.093 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:28:19.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:19.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:19.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:19.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:19.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.427 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:19.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:19.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:19.562 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:19.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: pgmap v376: 158 pgs: 158 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 782 op/s 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 8.f scrub starts 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:28:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: pgmap v376: 158 pgs: 158 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 782 op/s 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 8.f scrub starts 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:28:19.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:19 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:28:19.711 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:19.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:19.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:19.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: pgmap v376: 158 pgs: 158 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 782 op/s 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 8.f scrub starts 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:28:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:19 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:28:19.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:19.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:19.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:20.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:20.131 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:20.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:20.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:20.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:20.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:20.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:20.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:20.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:20.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:20.542 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:20.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:20.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:20.668 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:28:20.668 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:28:20.668 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 8.f scrub ok 2024-03-24T11:28:20.668 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:28:20.668 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 4.0 scrub starts 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 4.0 scrub ok 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 8.f scrub ok 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:28:20.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 4.0 scrub starts 2024-03-24T11:28:20.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 4.0 scrub ok 2024-03-24T11:28:20.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:28:20.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:28:20.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:28:20.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:28:20.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:28:20.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:20 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:28:20.684 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:20.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:20.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:20.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:20.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:28:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:28:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 8.f scrub ok 2024-03-24T11:28:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:28:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 4.0 scrub starts 2024-03-24T11:28:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 4.0 scrub ok 2024-03-24T11:28:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:28:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:28:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:28:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:28:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:28:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:20 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:28:20.922 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:20.957 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:20.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:21.021 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:21.046 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-03-24T11:28:18.198390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1568217462,"num_objects":24119,"num_object_clones":0,"num_object_copies":72357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24119,"num_whiteouts":0,"num_read":1279,"num_read_kb":1031,"num_write":232595,"num_write_kb":8194542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82903,"ondisk_log_size":82903,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6791124,"kb_used_data":4546272,"kb_used_omap":12,"kb_used_meta":2244275,"kb_avail":743170092,"statfs":{"total":767960285184,"available":761006174208,"internally_reserved":0,"allocated":4655382528,"data_stored":4653002090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2298138161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,10,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-409468928,"num_objects":-6248,"num_object_clones":0,"num_object_copies":-18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6248,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6248,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002492"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":288,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185474+0000","last_change":"2024-03-24T11:26:32.919289+0000","last_active":"2024-03-24T11:28:15.185474+0000","last_peered":"2024-03-24T11:28:15.185474+0000","last_clean":"2024-03-24T11:28:15.185474+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:28:15.185474+0000","last_undegraded":"2024-03-24T11:28:15.185474+0000","last_fullsized":"2024-03-24T11:28:15.185474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.919158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.919158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.388292+0000","last_change":"2024-03-24T11:27:17.030232+0000","last_active":"2024-03-24T11:28:11.388292+0000","last_peered":"2024-03-24T11:28:11.388292+0000","last_clean":"2024-03-24T11:28:11.388292+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:28:11.388292+0000","last_undegraded":"2024-03-24T11:28:11.388292+0000","last_fullsized":"2024-03-24T11:28:11.388292+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.030061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.030061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:23.838953+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.396473+0000","last_change":"2024-03-24T11:27:33.388210+0000","last_active":"2024-03-24T11:28:11.396473+0000","last_peered":"2024-03-24T11:28:11.396473+0000","last_clean":"2024-03-24T11:28:11.396473+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:28:11.396473+0000","last_undegraded":"2024-03-24T11:28:11.396473+0000","last_fullsized":"2024-03-24T11:28:11.396473+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:33.388096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:33.388096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:48.767655+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":336,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.183368+0000","last_change":"2024-03-24T11:28:15.009354+0000","last_active":"2024-03-24T11:28:15.183368+0000","last_peered":"2024-03-24T11:28:15.183368+0000","last_clean":"2024-03-24T11:28:15.183368+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:28:15.183368+0000","last_undegraded":"2024-03-24T11:28:15.183368+0000","last_fullsized":"2024-03-24T11:28:15.183368+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:28:15.009250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:28:15.009250+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:37.018757+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":270,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391108+0000","last_change":"2024-03-24T11:25:38.404738+0000","last_active":"2024-03-24T11:28:11.391108+0000","last_peered":"2024-03-24T11:28:11.391108+0000","last_clean":"2024-03-24T11:28:11.391108+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:28:11.391108+0000","last_undegraded":"2024-03-24T11:28:11.391108+0000","last_fullsized":"2024-03-24T11:28:11.391108+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:38.404511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:25:38.404511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":292,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.183467+0000","last_change":"2024-03-24T11:27:11.705436+0000","last_active":"2024-03-24T11:28:15.183467+0000","last_peered":"2024-03-24T11:28:15.183467+0000","last_clean":"2024-03-24T11:28:15.183467+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:28:15.183467+0000","last_undegraded":"2024-03-24T11:28:15.183467+0000","last_fullsized":"2024-03-24T11:28:15.183467+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:11.705334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.705334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.113548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393617+0000","last_change":"2024-03-24T11:27:31.498691+0000","last_active":"2024-03-24T11:28:11.393617+0000","last_peered":"2024-03-24T11:28:11.393617+0000","last_clean":"2024-03-24T11:28:11.393617+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:28:11.393617+0000","last_undegraded":"2024-03-24T11:28:11.393617+0000","last_fullsized":"2024-03-24T11:28:11.393617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:31.498583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.498583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:00.354290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":303,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393441+0000","last_change":"2024-03-24T11:26:29.288367+0000","last_active":"2024-03-24T11:28:11.393441+0000","last_peered":"2024-03-24T11:28:11.393441+0000","last_clean":"2024-03-24T11:28:11.393441+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:28:11.393441+0000","last_undegraded":"2024-03-24T11:28:11.393441+0000","last_fullsized":"2024-03-24T11:28:11.393441+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:26:29.288223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.288223+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.396381+0000","last_change":"2024-03-24T11:27:14.464091+0000","last_active":"2024-03-24T11:28:11.396381+0000","last_peered":"2024-03-24T11:28:11.396381+0000","last_clean":"2024-03-24T11:28:11.396381+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:28:11.396381+0000","last_undegraded":"2024-03-24T11:28:11.396381+0000","last_fullsized":"2024-03-24T11:28:11.396381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:14.463973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:14.463973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:16.251800+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.389424+0000","last_change":"2024-03-24T11:27:32.894325+0000","last_active":"2024-03-24T11:28:11.389424+0000","last_peered":"2024-03-24T11:28:11.389424+0000","last_clean":"2024-03-24T11:28:11.389424+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:28:11.389424+0000","last_undegraded":"2024-03-24T11:28:11.389424+0000","last_fullsized":"2024-03-24T11:28:11.389424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:27:32.894112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:01.062230+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393106+0000","last_change":"2024-03-24T11:25:46.067948+0000","last_active":"2024-03-24T11:28:11.393106+0000","last_peered":"2024-03-24T11:28:11.393106+0000","last_clean":"2024-03-24T11:28:11.393106+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:28:11.393106+0000","last_undegraded":"2024-03-24T11:28:11.393106+0000","last_fullsized":"2024-03-24T11:28:11.393106+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:25:46.067824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":312,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185638+0000","last_change":"2024-03-24T11:27:29.941329+0000","last_active":"2024-03-24T11:28:15.185638+0000","last_peered":"2024-03-24T11:28:15.185638+0000","last_clean":"2024-03-24T11:28:15.185638+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:28:15.185638+0000","last_undegraded":"2024-03-24T11:28:15.185638+0000","last_fullsized":"2024-03-24T11:28:15.185638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:27:29.941182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:29.941182+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:44.084844+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393902+0000","last_change":"2024-03-24T11:26:42.053183+0000","last_active":"2024-03-24T11:28:11.393902+0000","last_peered":"2024-03-24T11:28:11.393902+0000","last_clean":"2024-03-24T11:28:11.393902+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:28:11.393902+0000","last_undegraded":"2024-03-24T11:28:11.393902+0000","last_fullsized":"2024-03-24T11:28:11.393902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.052777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.052777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":292,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184975+0000","last_change":"2024-03-24T11:27:34.094930+0000","last_active":"2024-03-24T11:28:15.184975+0000","last_peered":"2024-03-24T11:28:15.184975+0000","last_clean":"2024-03-24T11:28:15.184975+0000","last_became_active":"2024-03-24T11:18:57.700375+0000","last_became_peered":"2024-03-24T11:18:57.700375+0000","last_unstale":"2024-03-24T11:28:15.184975+0000","last_undegraded":"2024-03-24T11:28:15.184975+0000","last_fullsized":"2024-03-24T11:28:15.184975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:34.094788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:27:34.094788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:53.948199+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391893+0000","last_change":"2024-03-24T11:26:43.090579+0000","last_active":"2024-03-24T11:28:11.391893+0000","last_peered":"2024-03-24T11:28:11.391893+0000","last_clean":"2024-03-24T11:28:11.391893+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:28:11.391893+0000","last_undegraded":"2024-03-24T11:28:11.391893+0000","last_fullsized":"2024-03-24T11:28:11.391893+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:43.090367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.090367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.868167+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":306,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185946+0000","last_change":"2024-03-24T11:26:47.926021+0000","last_active":"2024-03-24T11:28:15.185946+0000","last_peered":"2024-03-24T11:28:15.185946+0000","last_clean":"2024-03-24T11:28:15.185946+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:28:15.185946+0000","last_undegraded":"2024-03-24T11:28:15.185946+0000","last_fullsized":"2024-03-24T11:28:15.185946+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-24T11:26:47.925773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.925773+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":289,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186697+0000","last_change":"2024-03-24T11:26:27.996936+0000","last_active":"2024-03-24T11:28:15.186697+0000","last_peered":"2024-03-24T11:28:15.186697+0000","last_clean":"2024-03-24T11:28:15.186697+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:28:15.186697+0000","last_undegraded":"2024-03-24T11:28:15.186697+0000","last_fullsized":"2024-03-24T11:28:15.186697+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:27.996815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:27.996815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.389048+0000","last_change":"2024-03-24T11:27:18.078096+0000","last_active":"2024-03-24T11:28:11.389048+0000","last_peered":"2024-03-24T11:28:11.389048+0000","last_clean":"2024-03-24T11:28:11.389048+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:28:11.389048+0000","last_undegraded":"2024-03-24T11:28:11.389048+0000","last_fullsized":"2024-03-24T11:28:11.389048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:18.077961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.077961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:38.436426+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.397469+0000","last_change":"2024-03-24T11:26:32.451559+0000","last_active":"2024-03-24T11:28:11.397469+0000","last_peered":"2024-03-24T11:28:11.397469+0000","last_clean":"2024-03-24T11:28:11.397469+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:28:11.397469+0000","last_undegraded":"2024-03-24T11:28:11.397469+0000","last_fullsized":"2024-03-24T11:28:11.397469+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:32.451446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.451446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.282251+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":327,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393021+0000","last_change":"2024-03-24T11:27:22.975616+0000","last_active":"2024-03-24T11:28:11.393021+0000","last_peered":"2024-03-24T11:28:11.393021+0000","last_clean":"2024-03-24T11:28:11.393021+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:28:11.393021+0000","last_undegraded":"2024-03-24T11:28:11.393021+0000","last_fullsized":"2024-03-24T11:28:11.393021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-24T11:27:22.975468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:22.975468+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.206985+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.394157+0000","last_change":"2024-03-24T11:27:37.602861+0000","last_active":"2024-03-24T11:28:11.394157+0000","last_peered":"2024-03-24T11:28:11.394157+0000","last_clean":"2024-03-24T11:28:11.394157+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:28:11.394157+0000","last_undegraded":"2024-03-24T11:28:11.394157+0000","last_fullsized":"2024-03-24T11:28:11.394157+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:37.602750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:37.602750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.113684+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":300,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186527+0000","last_change":"2024-03-24T11:28:12.009635+0000","last_active":"2024-03-24T11:28:15.186527+0000","last_peered":"2024-03-24T11:28:15.186527+0000","last_clean":"2024-03-24T11:28:15.186527+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:28:15.186527+0000","last_undegraded":"2024-03-24T11:28:15.186527+0000","last_fullsized":"2024-03-24T11:28:15.186527+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:12.009510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:28:12.009510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:13.338455+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":300,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185735+0000","last_change":"2024-03-24T11:27:28.920285+0000","last_active":"2024-03-24T11:28:15.185735+0000","last_peered":"2024-03-24T11:28:15.185735+0000","last_clean":"2024-03-24T11:28:15.185735+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:28:15.185735+0000","last_undegraded":"2024-03-24T11:28:15.185735+0000","last_fullsized":"2024-03-24T11:28:15.185735+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:28.920173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:28.920173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.885240+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390878+0000","last_change":"2024-03-24T11:25:39.305828+0000","last_active":"2024-03-24T11:28:11.390878+0000","last_peered":"2024-03-24T11:28:11.390878+0000","last_clean":"2024-03-24T11:28:11.390878+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:28:11.390878+0000","last_undegraded":"2024-03-24T11:28:11.390878+0000","last_fullsized":"2024-03-24T11:28:11.390878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:25:39.305712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:39.305712+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":290,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185677+0000","last_change":"2024-03-24T11:27:17.910132+0000","last_active":"2024-03-24T11:28:15.185677+0000","last_peered":"2024-03-24T11:28:15.185677+0000","last_clean":"2024-03-24T11:28:15.185677+0000","last_became_active":"2024-03-24T11:18:59.707501+0000","last_became_peered":"2024-03-24T11:18:59.707501+0000","last_unstale":"2024-03-24T11:28:15.185677+0000","last_undegraded":"2024-03-24T11:28:15.185677+0000","last_fullsized":"2024-03-24T11:28:15.185677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:27:17.910050+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:20:22.871822+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.910050+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:26.378109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":426,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393739+0000","last_change":"2024-03-24T11:27:18.401734+0000","last_active":"2024-03-24T11:28:11.393739+0000","last_peered":"2024-03-24T11:28:11.393739+0000","last_clean":"2024-03-24T11:28:11.393739+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:28:11.393739+0000","last_undegraded":"2024-03-24T11:28:11.393739+0000","last_fullsized":"2024-03-24T11:28:11.393739+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:27:18.401578+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:23:53.129429+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.401578+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:27.954587+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":307,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.388373+0000","last_change":"2024-03-24T11:26:26.669460+0000","last_active":"2024-03-24T11:28:11.388373+0000","last_peered":"2024-03-24T11:28:11.388373+0000","last_clean":"2024-03-24T11:28:11.388373+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:28:11.388373+0000","last_undegraded":"2024-03-24T11:28:11.388373+0000","last_fullsized":"2024-03-24T11:28:11.388373+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:26:26.669216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.669216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.208600+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":318,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.403586+0000","last_change":"2024-03-24T11:27:23.363240+0000","last_active":"2024-03-24T11:28:11.403586+0000","last_peered":"2024-03-24T11:28:11.403586+0000","last_clean":"2024-03-24T11:28:11.403586+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:28:11.403586+0000","last_undegraded":"2024-03-24T11:28:11.403586+0000","last_fullsized":"2024-03-24T11:28:11.403586+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:27:23.362869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.362869+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:41.871796+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":294,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.389142+0000","last_change":"2024-03-24T11:27:24.109149+0000","last_active":"2024-03-24T11:28:11.389142+0000","last_peered":"2024-03-24T11:28:11.389142+0000","last_clean":"2024-03-24T11:28:11.389142+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:28:11.389142+0000","last_undegraded":"2024-03-24T11:28:11.389142+0000","last_fullsized":"2024-03-24T11:28:11.389142+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:24.108894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:58.788469+0000","last_clean_scrub_stamp":"2024-03-24T11:27:24.108894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:27.613843+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.403849+0000","last_change":"2024-03-24T11:26:31.491413+0000","last_active":"2024-03-24T11:28:11.403849+0000","last_peered":"2024-03-24T11:28:11.403849+0000","last_clean":"2024-03-24T11:28:11.403849+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:28:11.403849+0000","last_undegraded":"2024-03-24T11:28:11.403849+0000","last_fullsized":"2024-03-24T11:28:11.403849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:31.491292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:31.491292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.282366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391237+0000","last_change":"2024-03-24T11:26:26.240418+0000","last_active":"2024-03-24T11:28:11.391237+0000","last_peered":"2024-03-24T11:28:11.391237+0000","last_clean":"2024-03-24T11:28:11.391237+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:28:11.391237+0000","last_undegraded":"2024-03-24T11:28:11.391237+0000","last_fullsized":"2024-03-24T11:28:11.391237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.240302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.240302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"87'16","reported_seq":335,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393622+0000","last_change":"2024-03-24T11:27:23.466973+0000","last_active":"2024-03-24T11:28:11.393622+0000","last_peered":"2024-03-24T11:28:11.393622+0000","last_clean":"2024-03-24T11:28:11.393622+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:28:11.393622+0000","last_undegraded":"2024-03-24T11:28:11.393622+0000","last_fullsized":"2024-03-24T11:28:11.393622+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'16","last_scrub_stamp":"2024-03-24T11:27:23.466824+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.466824+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.025440+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":291,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184787+0000","last_change":"2024-03-24T11:27:20.015059+0000","last_active":"2024-03-24T11:28:15.184787+0000","last_peered":"2024-03-24T11:28:15.184787+0000","last_clean":"2024-03-24T11:28:15.184787+0000","last_became_active":"2024-03-24T11:18:59.707285+0000","last_became_peered":"2024-03-24T11:18:59.707285+0000","last_unstale":"2024-03-24T11:28:15.184787+0000","last_undegraded":"2024-03-24T11:28:15.184787+0000","last_fullsized":"2024-03-24T11:28:15.184787+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:20.014981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:20.014981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.746829+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390172+0000","last_change":"2024-03-24T11:26:47.560502+0000","last_active":"2024-03-24T11:28:11.390172+0000","last_peered":"2024-03-24T11:28:11.390172+0000","last_clean":"2024-03-24T11:28:11.390172+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:28:11.390172+0000","last_undegraded":"2024-03-24T11:28:11.390172+0000","last_fullsized":"2024-03-24T11:28:11.390172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:47.560381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:47.560381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.640940+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.396853+0000","last_change":"2024-03-24T11:27:43.240516+0000","last_active":"2024-03-24T11:28:11.396853+0000","last_peered":"2024-03-24T11:28:11.396853+0000","last_clean":"2024-03-24T11:28:11.396853+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:28:11.396853+0000","last_undegraded":"2024-03-24T11:28:11.396853+0000","last_fullsized":"2024-03-24T11:28:11.396853+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:43.240401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:43.240401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.243033+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"87'6","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391399+0000","last_change":"2024-03-24T11:27:13.458460+0000","last_active":"2024-03-24T11:28:11.391399+0000","last_peered":"2024-03-24T11:28:11.391399+0000","last_clean":"2024-03-24T11:28:11.391399+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:28:11.391399+0000","last_undegraded":"2024-03-24T11:28:11.391399+0000","last_fullsized":"2024-03-24T11:28:11.391399+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'6","last_scrub_stamp":"2024-03-24T11:27:13.458330+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:27:13.458330+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.492165+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391228+0000","last_change":"2024-03-24T11:27:41.544856+0000","last_active":"2024-03-24T11:28:11.391228+0000","last_peered":"2024-03-24T11:28:11.391228+0000","last_clean":"2024-03-24T11:28:11.391228+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:28:11.391228+0000","last_undegraded":"2024-03-24T11:28:11.391228+0000","last_fullsized":"2024-03-24T11:28:11.391228+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:41.544721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:41.544721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:03.082435+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":293,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184908+0000","last_change":"2024-03-24T11:27:33.054622+0000","last_active":"2024-03-24T11:28:15.184908+0000","last_peered":"2024-03-24T11:28:15.184908+0000","last_clean":"2024-03-24T11:28:15.184908+0000","last_became_active":"2024-03-24T11:18:57.698206+0000","last_became_peered":"2024-03-24T11:18:57.698206+0000","last_unstale":"2024-03-24T11:28:15.184908+0000","last_undegraded":"2024-03-24T11:28:15.184908+0000","last_fullsized":"2024-03-24T11:28:15.184908+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:33.054503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:33.054503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:02.259387+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.389464+0000","last_change":"2024-03-24T11:26:30.097875+0000","last_active":"2024-03-24T11:28:11.389464+0000","last_peered":"2024-03-24T11:28:11.389464+0000","last_clean":"2024-03-24T11:28:11.389464+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:28:11.389464+0000","last_undegraded":"2024-03-24T11:28:11.389464+0000","last_fullsized":"2024-03-24T11:28:11.389464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:30.097693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:30.097693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391582+0000","last_change":"2024-03-24T11:27:38.496836+0000","last_active":"2024-03-24T11:28:11.391582+0000","last_peered":"2024-03-24T11:28:11.391582+0000","last_clean":"2024-03-24T11:28:11.391582+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:28:11.391582+0000","last_undegraded":"2024-03-24T11:28:11.391582+0000","last_fullsized":"2024-03-24T11:28:11.391582+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:27:38.496690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:38.496690+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:52.289415+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.396000+0000","last_change":"2024-03-24T11:27:26.375854+0000","last_active":"2024-03-24T11:28:11.396000+0000","last_peered":"2024-03-24T11:28:11.396000+0000","last_clean":"2024-03-24T11:28:11.396000+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:28:11.396000+0000","last_undegraded":"2024-03-24T11:28:11.396000+0000","last_fullsized":"2024-03-24T11:28:11.396000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:26.375744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:26.375744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.913832+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391028+0000","last_change":"2024-03-24T11:27:12.419817+0000","last_active":"2024-03-24T11:28:11.391028+0000","last_peered":"2024-03-24T11:28:11.391028+0000","last_clean":"2024-03-24T11:28:11.391028+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:28:11.391028+0000","last_undegraded":"2024-03-24T11:28:11.391028+0000","last_fullsized":"2024-03-24T11:28:11.391028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:12.419688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:12.419688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.788280+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":300,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185629+0000","last_change":"2024-03-24T11:27:32.055899+0000","last_active":"2024-03-24T11:28:15.185629+0000","last_peered":"2024-03-24T11:28:15.185629+0000","last_clean":"2024-03-24T11:28:15.185629+0000","last_became_active":"2024-03-24T11:18:53.681374+0000","last_became_peered":"2024-03-24T11:18:53.681374+0000","last_unstale":"2024-03-24T11:28:15.185629+0000","last_undegraded":"2024-03-24T11:28:15.185629+0000","last_fullsized":"2024-03-24T11:28:15.185629+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:32.055790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:32.055790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:00.944756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":318,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393264+0000","last_change":"2024-03-24T11:27:19.050325+0000","last_active":"2024-03-24T11:28:11.393264+0000","last_peered":"2024-03-24T11:28:11.393264+0000","last_clean":"2024-03-24T11:28:11.393264+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:28:11.393264+0000","last_undegraded":"2024-03-24T11:28:11.393264+0000","last_fullsized":"2024-03-24T11:28:11.393264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:27:19.050122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:19.050122+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.462032+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390954+0000","last_change":"2024-03-24T11:26:33.238031+0000","last_active":"2024-03-24T11:28:11.390954+0000","last_peered":"2024-03-24T11:28:11.390954+0000","last_clean":"2024-03-24T11:28:11.390954+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:28:11.390954+0000","last_undegraded":"2024-03-24T11:28:11.390954+0000","last_fullsized":"2024-03-24T11:28:11.390954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.237864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.237864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":306,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393088+0000","last_change":"2024-03-24T11:27:28.018429+0000","last_active":"2024-03-24T11:28:11.393088+0000","last_peered":"2024-03-24T11:28:11.393088+0000","last_clean":"2024-03-24T11:28:11.393088+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:28:11.393088+0000","last_undegraded":"2024-03-24T11:28:11.393088+0000","last_fullsized":"2024-03-24T11:28:11.393088+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:27:28.018183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:49.907474+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":301,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184266+0000","last_change":"2024-03-24T11:26:33.874445+0000","last_active":"2024-03-24T11:28:15.184266+0000","last_peered":"2024-03-24T11:28:15.184266+0000","last_clean":"2024-03-24T11:28:15.184266+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:28:15.184266+0000","last_undegraded":"2024-03-24T11:28:15.184266+0000","last_fullsized":"2024-03-24T11:28:15.184266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:33.874331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:26:33.874331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"129'16","reported_seq":329,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186234+0000","last_change":"2024-03-24T11:27:39.010991+0000","last_active":"2024-03-24T11:28:15.186234+0000","last_peered":"2024-03-24T11:28:15.186234+0000","last_clean":"2024-03-24T11:28:15.186234+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:28:15.186234+0000","last_undegraded":"2024-03-24T11:28:15.186234+0000","last_fullsized":"2024-03-24T11:28:15.186234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'16","last_scrub_stamp":"2024-03-24T11:27:39.010853+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:27:39.010853+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:43.107310+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":286,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.402383+0000","last_change":"2024-03-24T11:27:31.452484+0000","last_active":"2024-03-24T11:28:11.402383+0000","last_peered":"2024-03-24T11:28:11.402383+0000","last_clean":"2024-03-24T11:28:11.402383+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:28:11.402383+0000","last_undegraded":"2024-03-24T11:28:11.402383+0000","last_fullsized":"2024-03-24T11:28:11.402383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:31.452376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.452376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:47.520811+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":418,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393220+0000","last_change":"2024-03-24T11:28:09.838066+0000","last_active":"2024-03-24T11:28:11.393220+0000","last_peered":"2024-03-24T11:28:11.393220+0000","last_clean":"2024-03-24T11:28:11.393220+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:28:11.393220+0000","last_undegraded":"2024-03-24T11:28:11.393220+0000","last_fullsized":"2024-03-24T11:28:11.393220+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:28:09.837980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:28:09.837980+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:25.975130+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":301,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184609+0000","last_change":"2024-03-24T11:27:35.074782+0000","last_active":"2024-03-24T11:28:15.184609+0000","last_peered":"2024-03-24T11:28:15.184609+0000","last_clean":"2024-03-24T11:28:15.184609+0000","last_became_active":"2024-03-24T11:18:53.681857+0000","last_became_peered":"2024-03-24T11:18:53.681857+0000","last_unstale":"2024-03-24T11:28:15.184609+0000","last_undegraded":"2024-03-24T11:28:15.184609+0000","last_fullsized":"2024-03-24T11:28:15.184609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:35.074657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:35.074657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:35.542279+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":301,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391100+0000","last_change":"2024-03-24T11:27:24.533619+0000","last_active":"2024-03-24T11:28:11.391100+0000","last_peered":"2024-03-24T11:28:11.391100+0000","last_clean":"2024-03-24T11:28:11.391100+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:28:11.391100+0000","last_undegraded":"2024-03-24T11:28:11.391100+0000","last_fullsized":"2024-03-24T11:28:11.391100+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:27:24.533484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:24.533484+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.684360+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":288,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184308+0000","last_change":"2024-03-24T11:26:39.812393+0000","last_active":"2024-03-24T11:28:15.184308+0000","last_peered":"2024-03-24T11:28:15.184308+0000","last_clean":"2024-03-24T11:28:15.184308+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:28:15.184308+0000","last_undegraded":"2024-03-24T11:28:15.184308+0000","last_fullsized":"2024-03-24T11:28:15.184308+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.812259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":402,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391096+0000","last_change":"2024-03-24T11:25:49.322508+0000","last_active":"2024-03-24T11:28:11.391096+0000","last_peered":"2024-03-24T11:28:11.391096+0000","last_clean":"2024-03-24T11:28:11.391096+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:28:11.391096+0000","last_undegraded":"2024-03-24T11:28:11.391096+0000","last_fullsized":"2024-03-24T11:28:11.391096+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:49.322383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:49.322383+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.398712+0000","last_change":"2024-03-24T11:26:46.640485+0000","last_active":"2024-03-24T11:28:11.398712+0000","last_peered":"2024-03-24T11:28:11.398712+0000","last_clean":"2024-03-24T11:28:11.398712+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:28:11.398712+0000","last_undegraded":"2024-03-24T11:28:11.398712+0000","last_fullsized":"2024-03-24T11:28:11.398712+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.640304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.282338+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":318,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393450+0000","last_change":"2024-03-24T11:27:39.856707+0000","last_active":"2024-03-24T11:28:11.393450+0000","last_peered":"2024-03-24T11:28:11.393450+0000","last_clean":"2024-03-24T11:28:11.393450+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:28:11.393450+0000","last_undegraded":"2024-03-24T11:28:11.393450+0000","last_fullsized":"2024-03-24T11:28:11.393450+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:27:39.856583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:39.856583+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:50.163370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":302,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:12.131852+0000","last_change":"2024-03-24T11:28:12.131893+0000","last_active":"2024-03-24T11:28:12.131852+0000","last_peered":"2024-03-24T11:28:12.131852+0000","last_clean":"2024-03-24T11:28:12.131852+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:28:12.131852+0000","last_undegraded":"2024-03-24T11:28:12.131852+0000","last_fullsized":"2024-03-24T11:28:12.131852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:12.131779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:28:12.131779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:36.833728+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.394492+0000","last_change":"2024-03-24T11:26:42.302143+0000","last_active":"2024-03-24T11:28:11.394492+0000","last_peered":"2024-03-24T11:28:11.394492+0000","last_clean":"2024-03-24T11:28:11.394492+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:28:11.394492+0000","last_undegraded":"2024-03-24T11:28:11.394492+0000","last_fullsized":"2024-03-24T11:28:11.394492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.301950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":309,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184774+0000","last_change":"2024-03-24T11:28:13.026000+0000","last_active":"2024-03-24T11:28:15.184774+0000","last_peered":"2024-03-24T11:28:15.184774+0000","last_clean":"2024-03-24T11:28:15.184774+0000","last_became_active":"2024-03-24T11:18:53.679981+0000","last_became_peered":"2024-03-24T11:18:53.679981+0000","last_unstale":"2024-03-24T11:28:15.184774+0000","last_undegraded":"2024-03-24T11:28:15.184774+0000","last_fullsized":"2024-03-24T11:28:15.184774+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:13.025883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:28:13.025883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:31.468328+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"86'13","reported_seq":322,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.403688+0000","last_change":"2024-03-24T11:27:38.277566+0000","last_active":"2024-03-24T11:28:11.403688+0000","last_peered":"2024-03-24T11:28:11.403688+0000","last_clean":"2024-03-24T11:28:11.403688+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:28:11.403688+0000","last_undegraded":"2024-03-24T11:28:11.403688+0000","last_fullsized":"2024-03-24T11:28:11.403688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-03-24T11:27:38.277150+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:27:38.277150+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.877816+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":288,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185471+0000","last_change":"2024-03-24T11:27:31.087000+0000","last_active":"2024-03-24T11:28:15.185471+0000","last_peered":"2024-03-24T11:28:15.185471+0000","last_clean":"2024-03-24T11:28:15.185471+0000","last_became_active":"2024-03-24T11:19:00.314143+0000","last_became_peered":"2024-03-24T11:19:00.314143+0000","last_unstale":"2024-03-24T11:28:15.185471+0000","last_undegraded":"2024-03-24T11:28:15.185471+0000","last_fullsized":"2024-03-24T11:28:15.185471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:31.086873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:31.086873+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.086873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:58.266470+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.914401+0000","last_change":"2024-03-24T11:28:11.914439+0000","last_active":"2024-03-24T11:28:11.914401+0000","last_peered":"2024-03-24T11:28:11.914401+0000","last_clean":"2024-03-24T11:28:11.914401+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:28:11.914401+0000","last_undegraded":"2024-03-24T11:28:11.914401+0000","last_fullsized":"2024-03-24T11:28:11.914401+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:11.914321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:28:11.914321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:34.831571+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":315,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.653196+0000","last_change":"2024-03-24T11:28:11.653196+0000","last_active":"2024-03-24T11:28:11.653196+0000","last_peered":"2024-03-24T11:28:11.653196+0000","last_clean":"2024-03-24T11:28:11.653196+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:28:11.653196+0000","last_undegraded":"2024-03-24T11:28:11.653196+0000","last_fullsized":"2024-03-24T11:28:11.653196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:11.653081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:28:11.653081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:21.145461+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"186'9","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.394583+0000","last_change":"2024-03-24T11:27:33.522192+0000","last_active":"2024-03-24T11:28:11.394583+0000","last_peered":"2024-03-24T11:28:11.394583+0000","last_clean":"2024-03-24T11:28:11.394583+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:28:11.394583+0000","last_undegraded":"2024-03-24T11:28:11.394583+0000","last_fullsized":"2024-03-24T11:28:11.394583+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:27:33.522057+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:55.458927+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"192'14479","reported_seq":14632,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.207899+0000","last_change":"2024-03-24T11:27:27.279069+0000","last_active":"2024-03-24T11:28:13.207899+0000","last_peered":"2024-03-24T11:28:13.207899+0000","last_clean":"2024-03-24T11:28:13.207899+0000","last_became_active":"2024-03-24T11:25:02.676873+0000","last_became_peered":"2024-03-24T11:25:02.676873+0000","last_unstale":"2024-03-24T11:28:13.207899+0000","last_undegraded":"2024-03-24T11:28:13.207899+0000","last_fullsized":"2024-03-24T11:28:13.207899+0000","mapping_epoch":150,"log_start":"180'9400","ondisk_log_start":"180'9400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'11080","last_scrub_stamp":"2024-03-24T11:27:27.278930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.278930+0000","objects_scrubbed":5005,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:47.009881+0000","scrub_duration":1305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":4671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14479,"num_write_kb":513152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393995+0000","last_change":"2024-03-24T11:27:36.870198+0000","last_active":"2024-03-24T11:28:11.393995+0000","last_peered":"2024-03-24T11:28:11.393995+0000","last_clean":"2024-03-24T11:28:11.393995+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:28:11.393995+0000","last_undegraded":"2024-03-24T11:28:11.393995+0000","last_fullsized":"2024-03-24T11:28:11.393995+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:36.870117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:36.870117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:58.238790+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":323,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:17.880965+0000","last_change":"2024-03-24T11:28:17.881008+0000","last_active":"2024-03-24T11:28:17.880965+0000","last_peered":"2024-03-24T11:28:17.880965+0000","last_clean":"2024-03-24T11:28:17.880965+0000","last_became_active":"2024-03-24T11:18:53.684322+0000","last_became_peered":"2024-03-24T11:18:53.684322+0000","last_unstale":"2024-03-24T11:28:17.880965+0000","last_undegraded":"2024-03-24T11:28:17.880965+0000","last_fullsized":"2024-03-24T11:28:17.880965+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:28:17.880895+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:28:17.880895+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:27.940999+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"129'22","reported_seq":335,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391040+0000","last_change":"2024-03-24T11:27:37.844432+0000","last_active":"2024-03-24T11:28:11.391040+0000","last_peered":"2024-03-24T11:28:11.391040+0000","last_clean":"2024-03-24T11:28:11.391040+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:28:11.391040+0000","last_undegraded":"2024-03-24T11:28:11.391040+0000","last_fullsized":"2024-03-24T11:28:11.391040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'22","last_scrub_stamp":"2024-03-24T11:27:37.844238+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:27:37.844238+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.737653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":414,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185014+0000","last_change":"2024-03-24T11:26:48.966305+0000","last_active":"2024-03-24T11:28:15.185014+0000","last_peered":"2024-03-24T11:28:15.185014+0000","last_clean":"2024-03-24T11:28:15.185014+0000","last_became_active":"2024-03-24T11:18:57.700136+0000","last_became_peered":"2024-03-24T11:18:57.700136+0000","last_unstale":"2024-03-24T11:28:15.185014+0000","last_undegraded":"2024-03-24T11:28:15.185014+0000","last_fullsized":"2024-03-24T11:28:15.185014+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:48.966187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.966187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.360272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"192'14523","reported_seq":14705,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.184221+0000","last_change":"2024-03-24T11:26:46.326986+0000","last_active":"2024-03-24T11:28:13.184221+0000","last_peered":"2024-03-24T11:28:13.184221+0000","last_clean":"2024-03-24T11:28:13.184221+0000","last_became_active":"2024-03-24T11:25:02.675760+0000","last_became_peered":"2024-03-24T11:25:02.675760+0000","last_unstale":"2024-03-24T11:28:13.184221+0000","last_undegraded":"2024-03-24T11:28:13.184221+0000","last_fullsized":"2024-03-24T11:28:13.184221+0000","mapping_epoch":150,"log_start":"179'8700","ondisk_log_start":"179'8700","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'8169","last_scrub_stamp":"2024-03-24T11:26:46.326655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.326655+0000","objects_scrubbed":7921,"log_size":5823,"log_dups_size":0,"ondisk_log_size":5823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99287040,"num_objects":1515,"num_object_clones":0,"num_object_copies":4545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14523,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391183+0000","last_change":"2024-03-24T11:27:26.540096+0000","last_active":"2024-03-24T11:28:11.391183+0000","last_peered":"2024-03-24T11:28:11.391183+0000","last_clean":"2024-03-24T11:28:11.391183+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:28:11.391183+0000","last_undegraded":"2024-03-24T11:28:11.391183+0000","last_fullsized":"2024-03-24T11:28:11.391183+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:26.539694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:26.539694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.570356+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":301,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185586+0000","last_change":"2024-03-24T11:28:11.023211+0000","last_active":"2024-03-24T11:28:15.185586+0000","last_peered":"2024-03-24T11:28:15.185586+0000","last_clean":"2024-03-24T11:28:15.185586+0000","last_became_active":"2024-03-24T11:18:53.682464+0000","last_became_peered":"2024-03-24T11:18:53.682464+0000","last_unstale":"2024-03-24T11:28:15.185586+0000","last_undegraded":"2024-03-24T11:28:15.185586+0000","last_fullsized":"2024-03-24T11:28:15.185586+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:11.023099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:28:11.023099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:14.652169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":328,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.183986+0000","last_change":"2024-03-24T11:28:12.961236+0000","last_active":"2024-03-24T11:28:15.183986+0000","last_peered":"2024-03-24T11:28:15.183986+0000","last_clean":"2024-03-24T11:28:15.183986+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:28:15.183986+0000","last_undegraded":"2024-03-24T11:28:15.183986+0000","last_fullsized":"2024-03-24T11:28:15.183986+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:28:12.961099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:28:12.961099+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:41.033113+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.382859+0000","last_change":"2024-03-24T11:26:37.862505+0000","last_active":"2024-03-24T11:28:11.382859+0000","last_peered":"2024-03-24T11:28:11.382859+0000","last_clean":"2024-03-24T11:28:11.382859+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:28:11.382859+0000","last_undegraded":"2024-03-24T11:28:11.382859+0000","last_fullsized":"2024-03-24T11:28:11.382859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.862389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.862389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"192'14573","reported_seq":14731,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:14.063085+0000","last_change":"2024-03-24T11:27:18.597916+0000","last_active":"2024-03-24T11:28:14.063085+0000","last_peered":"2024-03-24T11:28:14.063085+0000","last_clean":"2024-03-24T11:28:14.063085+0000","last_became_active":"2024-03-24T11:25:02.676144+0000","last_became_peered":"2024-03-24T11:25:02.676144+0000","last_unstale":"2024-03-24T11:28:14.063085+0000","last_undegraded":"2024-03-24T11:28:14.063085+0000","last_fullsized":"2024-03-24T11:28:14.063085+0000","mapping_epoch":150,"log_start":"179'9100","ondisk_log_start":"179'9100","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10592","last_scrub_stamp":"2024-03-24T11:27:18.597618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.597618+0000","objects_scrubbed":5591,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.670378+0000","scrub_duration":3172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97845248,"num_objects":1493,"num_object_clones":0,"num_object_copies":4479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14573,"num_write_kb":514112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":289,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184390+0000","last_change":"2024-03-24T11:28:12.009417+0000","last_active":"2024-03-24T11:28:15.184390+0000","last_peered":"2024-03-24T11:28:15.184390+0000","last_clean":"2024-03-24T11:28:15.184390+0000","last_became_active":"2024-03-24T11:18:59.706981+0000","last_became_peered":"2024-03-24T11:18:59.706981+0000","last_unstale":"2024-03-24T11:28:15.184390+0000","last_undegraded":"2024-03-24T11:28:15.184390+0000","last_fullsized":"2024-03-24T11:28:15.184390+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:12.009306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:28:12.009306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:13.911786+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393312+0000","last_change":"2024-03-24T11:26:41.913283+0000","last_active":"2024-03-24T11:28:11.393312+0000","last_peered":"2024-03-24T11:28:11.393312+0000","last_clean":"2024-03-24T11:28:11.393312+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:28:11.393312+0000","last_undegraded":"2024-03-24T11:28:11.393312+0000","last_fullsized":"2024-03-24T11:28:11.393312+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.913165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.913165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":309,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:16.935388+0000","last_change":"2024-03-24T11:28:16.935428+0000","last_active":"2024-03-24T11:28:16.935388+0000","last_peered":"2024-03-24T11:28:16.935388+0000","last_clean":"2024-03-24T11:28:16.935388+0000","last_became_active":"2024-03-24T11:18:53.683333+0000","last_became_peered":"2024-03-24T11:18:53.683333+0000","last_unstale":"2024-03-24T11:28:16.935388+0000","last_undegraded":"2024-03-24T11:28:16.935388+0000","last_fullsized":"2024-03-24T11:28:16.935388+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:16.935308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:28:16.935308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:30.222394+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"88'28","reported_seq":343,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390190+0000","last_change":"2024-03-24T11:26:34.694013+0000","last_active":"2024-03-24T11:28:11.390190+0000","last_peered":"2024-03-24T11:28:11.390190+0000","last_clean":"2024-03-24T11:28:11.390190+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:28:11.390190+0000","last_undegraded":"2024-03-24T11:28:11.390190+0000","last_fullsized":"2024-03-24T11:28:11.390190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'28","last_scrub_stamp":"2024-03-24T11:26:34.693862+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.693862+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"192'14427","reported_seq":14609,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.526102+0000","last_change":"2024-03-24T11:26:44.358566+0000","last_active":"2024-03-24T11:28:13.526102+0000","last_peered":"2024-03-24T11:28:13.526102+0000","last_clean":"2024-03-24T11:28:13.526102+0000","last_became_active":"2024-03-24T11:25:02.599484+0000","last_became_peered":"2024-03-24T11:25:02.599484+0000","last_unstale":"2024-03-24T11:28:13.526102+0000","last_undegraded":"2024-03-24T11:28:13.526102+0000","last_fullsized":"2024-03-24T11:28:13.526102+0000","mapping_epoch":150,"log_start":"181'9500","ondisk_log_start":"181'9500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7990","last_scrub_stamp":"2024-03-24T11:26:44.358125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:44.358125+0000","objects_scrubbed":7882,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99942400,"num_objects":1525,"num_object_clones":0,"num_object_copies":4575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14427,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.398486+0000","last_change":"2024-03-24T11:26:26.644711+0000","last_active":"2024-03-24T11:28:11.398486+0000","last_peered":"2024-03-24T11:28:11.398486+0000","last_clean":"2024-03-24T11:28:11.398486+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:28:11.398486+0000","last_undegraded":"2024-03-24T11:28:11.398486+0000","last_fullsized":"2024-03-24T11:28:11.398486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:26.644597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:26.644597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.282309+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":307,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393339+0000","last_change":"2024-03-24T11:28:11.111162+0000","last_active":"2024-03-24T11:28:11.393339+0000","last_peered":"2024-03-24T11:28:11.393339+0000","last_clean":"2024-03-24T11:28:11.393339+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:28:11.393339+0000","last_undegraded":"2024-03-24T11:28:11.393339+0000","last_fullsized":"2024-03-24T11:28:11.393339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:11.110682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:28:11.110682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:40.522560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:12.947980+0000","last_change":"2024-03-24T11:28:12.948021+0000","last_active":"2024-03-24T11:28:12.947980+0000","last_peered":"2024-03-24T11:28:12.947980+0000","last_clean":"2024-03-24T11:28:12.947980+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:28:12.947980+0000","last_undegraded":"2024-03-24T11:28:12.947980+0000","last_fullsized":"2024-03-24T11:28:12.947980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:28:12.947895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:28:12.947895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:19.624541+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.397598+0000","last_change":"2024-03-24T11:26:29.507764+0000","last_active":"2024-03-24T11:28:11.397598+0000","last_peered":"2024-03-24T11:28:11.397598+0000","last_clean":"2024-03-24T11:28:11.397598+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:28:11.397598+0000","last_undegraded":"2024-03-24T11:28:11.397598+0000","last_fullsized":"2024-03-24T11:28:11.397598+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-24T11:26:29.507604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.507604+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.282218+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"192'14745","reported_seq":14910,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.217189+0000","last_change":"2024-03-24T11:27:15.950757+0000","last_active":"2024-03-24T11:28:13.217189+0000","last_peered":"2024-03-24T11:28:13.217189+0000","last_clean":"2024-03-24T11:28:13.217189+0000","last_became_active":"2024-03-24T11:25:02.675496+0000","last_became_peered":"2024-03-24T11:25:02.675496+0000","last_unstale":"2024-03-24T11:28:13.217189+0000","last_undegraded":"2024-03-24T11:28:13.217189+0000","last_fullsized":"2024-03-24T11:28:13.217189+0000","mapping_epoch":150,"log_start":"180'9700","ondisk_log_start":"180'9700","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10572","last_scrub_stamp":"2024-03-24T11:27:15.950518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:15.950518+0000","objects_scrubbed":5747,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.068187+0000","scrub_duration":1924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98631680,"num_objects":1505,"num_object_clones":0,"num_object_copies":4515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14745,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":429,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.394079+0000","last_change":"2024-03-24T11:25:48.067965+0000","last_active":"2024-03-24T11:28:11.394079+0000","last_peered":"2024-03-24T11:28:11.394079+0000","last_clean":"2024-03-24T11:28:11.394079+0000","last_became_active":"2024-03-24T11:24:29.093656+0000","last_became_peered":"2024-03-24T11:24:29.093656+0000","last_unstale":"2024-03-24T11:28:11.394079+0000","last_undegraded":"2024-03-24T11:28:11.394079+0000","last_fullsized":"2024-03-24T11:28:11.394079+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:48.067828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:48.067828+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393825+0000","last_change":"2024-03-24T11:26:40.231258+0000","last_active":"2024-03-24T11:28:11.393825+0000","last_peered":"2024-03-24T11:28:11.393825+0000","last_clean":"2024-03-24T11:28:11.393825+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:28:11.393825+0000","last_undegraded":"2024-03-24T11:28:11.393825+0000","last_fullsized":"2024-03-24T11:28:11.393825+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:40.231135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:26:40.231135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":310,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393403+0000","last_change":"2024-03-24T11:25:35.099894+0000","last_active":"2024-03-24T11:28:11.393403+0000","last_peered":"2024-03-24T11:28:11.393403+0000","last_clean":"2024-03-24T11:28:11.393403+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:28:11.393403+0000","last_undegraded":"2024-03-24T11:28:11.393403+0000","last_fullsized":"2024-03-24T11:28:11.393403+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:25:35.099746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.099746+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":309,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186011+0000","last_change":"2024-03-24T11:28:14.005806+0000","last_active":"2024-03-24T11:28:15.186011+0000","last_peered":"2024-03-24T11:28:15.186011+0000","last_clean":"2024-03-24T11:28:15.186011+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:28:15.186011+0000","last_undegraded":"2024-03-24T11:28:15.186011+0000","last_fullsized":"2024-03-24T11:28:15.186011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:14.005727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:28:14.005727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:25.439913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":463,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.392210+0000","last_change":"2024-03-24T11:27:13.958327+0000","last_active":"2024-03-24T11:28:11.392210+0000","last_peered":"2024-03-24T11:28:11.392210+0000","last_clean":"2024-03-24T11:28:11.392210+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:28:11.392210+0000","last_undegraded":"2024-03-24T11:28:11.392210+0000","last_fullsized":"2024-03-24T11:28:11.392210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-03-24T11:27:13.958181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:13.958181+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.681771+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.396401+0000","last_change":"2024-03-24T11:26:38.524661+0000","last_active":"2024-03-24T11:28:11.396401+0000","last_peered":"2024-03-24T11:28:11.396401+0000","last_clean":"2024-03-24T11:28:11.396401+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:28:11.396401+0000","last_undegraded":"2024-03-24T11:28:11.396401+0000","last_fullsized":"2024-03-24T11:28:11.396401+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:38.524541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:38.524541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.282394+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"192'14904","reported_seq":15029,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.736175+0000","last_change":"2024-03-24T11:28:13.057788+0000","last_active":"2024-03-24T11:28:13.736175+0000","last_peered":"2024-03-24T11:28:13.736175+0000","last_clean":"2024-03-24T11:28:13.736175+0000","last_became_active":"2024-03-24T11:25:02.600777+0000","last_became_peered":"2024-03-24T11:25:02.600777+0000","last_unstale":"2024-03-24T11:28:13.736175+0000","last_undegraded":"2024-03-24T11:28:13.736175+0000","last_fullsized":"2024-03-24T11:28:13.736175+0000","mapping_epoch":150,"log_start":"184'10400","ondisk_log_start":"184'10400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'14847","last_scrub_stamp":"2024-03-24T11:28:13.057573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:28:13.057573+0000","objects_scrubbed":1628,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:13.882786+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101580800,"num_objects":1550,"num_object_clones":0,"num_object_copies":4650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14904,"num_write_kb":526528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":579,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184878+0000","last_change":"2024-03-24T11:27:40.000582+0000","last_active":"2024-03-24T11:28:15.184878+0000","last_peered":"2024-03-24T11:28:15.184878+0000","last_clean":"2024-03-24T11:28:15.184878+0000","last_became_active":"2024-03-24T11:24:29.092038+0000","last_became_peered":"2024-03-24T11:24:29.092038+0000","last_unstale":"2024-03-24T11:28:15.184878+0000","last_undegraded":"2024-03-24T11:28:15.184878+0000","last_fullsized":"2024-03-24T11:28:15.184878+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:27:40.000450+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.673115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":297,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185897+0000","last_change":"2024-03-24T11:27:41.933217+0000","last_active":"2024-03-24T11:28:15.185897+0000","last_peered":"2024-03-24T11:28:15.185897+0000","last_clean":"2024-03-24T11:28:15.185897+0000","last_became_active":"2024-03-24T11:19:00.311560+0000","last_became_peered":"2024-03-24T11:19:00.311560+0000","last_unstale":"2024-03-24T11:28:15.185897+0000","last_undegraded":"2024-03-24T11:28:15.185897+0000","last_fullsized":"2024-03-24T11:28:15.185897+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:41.933084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:27:41.933084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:56.249244+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":315,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.388964+0000","last_change":"2024-03-24T11:27:34.042910+0000","last_active":"2024-03-24T11:28:11.388964+0000","last_peered":"2024-03-24T11:28:11.388964+0000","last_clean":"2024-03-24T11:28:11.388964+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:28:11.388964+0000","last_undegraded":"2024-03-24T11:28:11.388964+0000","last_fullsized":"2024-03-24T11:28:11.388964+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:34.042779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:34.042779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:48.561916+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":316,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.392907+0000","last_change":"2024-03-24T11:27:21.956869+0000","last_active":"2024-03-24T11:28:11.392907+0000","last_peered":"2024-03-24T11:28:11.392907+0000","last_clean":"2024-03-24T11:28:11.392907+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:28:11.392907+0000","last_undegraded":"2024-03-24T11:28:11.392907+0000","last_fullsized":"2024-03-24T11:28:11.392907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:21.956404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:27:21.956404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:37.958040+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393039+0000","last_change":"2024-03-24T11:27:23.937410+0000","last_active":"2024-03-24T11:28:11.393039+0000","last_peered":"2024-03-24T11:28:11.393039+0000","last_clean":"2024-03-24T11:28:11.393039+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:28:11.393039+0000","last_undegraded":"2024-03-24T11:28:11.393039+0000","last_fullsized":"2024-03-24T11:28:11.393039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-24T11:27:23.936937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.936937+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:43.630650+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":307,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.395155+0000","last_change":"2024-03-24T11:28:10.087231+0000","last_active":"2024-03-24T11:28:11.395155+0000","last_peered":"2024-03-24T11:28:11.395155+0000","last_clean":"2024-03-24T11:28:11.395155+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:28:11.395155+0000","last_undegraded":"2024-03-24T11:28:11.395155+0000","last_fullsized":"2024-03-24T11:28:11.395155+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:10.086930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:28:10.086930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:30.371823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"192'14184","reported_seq":14365,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.238134+0000","last_change":"2024-03-24T11:26:48.313995+0000","last_active":"2024-03-24T11:28:13.238134+0000","last_peered":"2024-03-24T11:28:13.238134+0000","last_clean":"2024-03-24T11:28:13.238134+0000","last_became_active":"2024-03-24T11:25:02.600137+0000","last_became_peered":"2024-03-24T11:25:02.600137+0000","last_unstale":"2024-03-24T11:28:13.238134+0000","last_undegraded":"2024-03-24T11:28:13.238134+0000","last_fullsized":"2024-03-24T11:28:13.238134+0000","mapping_epoch":150,"log_start":"179'8400","ondisk_log_start":"179'8400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8126","last_scrub_stamp":"2024-03-24T11:26:48.313801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.313801+0000","objects_scrubbed":7677,"log_size":5784,"log_dups_size":0,"ondisk_log_size":5784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100532224,"num_objects":1534,"num_object_clones":0,"num_object_copies":4602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14184,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.389651+0000","last_change":"2024-03-24T11:26:49.074595+0000","last_active":"2024-03-24T11:28:11.389651+0000","last_peered":"2024-03-24T11:28:11.389651+0000","last_clean":"2024-03-24T11:28:11.389651+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:28:11.389651+0000","last_undegraded":"2024-03-24T11:28:11.389651+0000","last_fullsized":"2024-03-24T11:28:11.389651+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.074485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":564,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184600+0000","last_change":"2024-03-24T11:26:36.872094+0000","last_active":"2024-03-24T11:28:15.184600+0000","last_peered":"2024-03-24T11:28:15.184600+0000","last_clean":"2024-03-24T11:28:15.184600+0000","last_became_active":"2024-03-24T11:24:23.650976+0000","last_became_peered":"2024-03-24T11:24:23.650976+0000","last_unstale":"2024-03-24T11:28:15.184600+0000","last_undegraded":"2024-03-24T11:28:15.184600+0000","last_fullsized":"2024-03-24T11:28:15.184600+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:36.871960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.871960+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":297,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390546+0000","last_change":"2024-03-24T11:27:12.329869+0000","last_active":"2024-03-24T11:28:11.390546+0000","last_peered":"2024-03-24T11:28:11.390546+0000","last_clean":"2024-03-24T11:28:11.390546+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:28:11.390546+0000","last_undegraded":"2024-03-24T11:28:11.390546+0000","last_fullsized":"2024-03-24T11:28:11.390546+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:35.227538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:35.227538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"176'24","reported_seq":453,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390797+0000","last_change":"2024-03-24T11:28:09.577166+0000","last_active":"2024-03-24T11:28:11.390797+0000","last_peered":"2024-03-24T11:28:11.390797+0000","last_clean":"2024-03-24T11:28:11.390797+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:28:11.390797+0000","last_undegraded":"2024-03-24T11:28:11.390797+0000","last_fullsized":"2024-03-24T11:28:11.390797+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'24","last_scrub_stamp":"2024-03-24T11:28:09.577035+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:28:09.577035+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:12.406083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":301,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185875+0000","last_change":"2024-03-24T11:27:11.941555+0000","last_active":"2024-03-24T11:28:15.185875+0000","last_peered":"2024-03-24T11:28:15.185875+0000","last_clean":"2024-03-24T11:28:15.185875+0000","last_became_active":"2024-03-24T11:18:57.699044+0000","last_became_peered":"2024-03-24T11:18:57.699044+0000","last_unstale":"2024-03-24T11:28:15.185875+0000","last_undegraded":"2024-03-24T11:28:15.185875+0000","last_fullsized":"2024-03-24T11:28:15.185875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:11.941414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.941414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.864708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"192'14347","reported_seq":14528,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.551861+0000","last_change":"2024-03-24T11:26:42.718381+0000","last_active":"2024-03-24T11:28:13.551861+0000","last_peered":"2024-03-24T11:28:13.551861+0000","last_clean":"2024-03-24T11:28:13.551861+0000","last_became_active":"2024-03-24T11:25:02.674959+0000","last_became_peered":"2024-03-24T11:25:02.674959+0000","last_unstale":"2024-03-24T11:28:13.551861+0000","last_undegraded":"2024-03-24T11:28:13.551861+0000","last_fullsized":"2024-03-24T11:28:13.551861+0000","mapping_epoch":150,"log_start":"181'9500","ondisk_log_start":"181'9500","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7858","last_scrub_stamp":"2024-03-24T11:26:42.718210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.718210+0000","objects_scrubbed":7774,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99418112,"num_objects":1517,"num_object_clones":0,"num_object_copies":4551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14347,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":437,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.388689+0000","last_change":"2024-03-24T11:27:31.970133+0000","last_active":"2024-03-24T11:28:11.388689+0000","last_peered":"2024-03-24T11:28:11.388689+0000","last_clean":"2024-03-24T11:28:11.388689+0000","last_became_active":"2024-03-24T11:24:29.087723+0000","last_became_peered":"2024-03-24T11:24:29.087723+0000","last_unstale":"2024-03-24T11:28:11.388689+0000","last_undegraded":"2024-03-24T11:28:11.388689+0000","last_fullsized":"2024-03-24T11:28:11.388689+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:31.969994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:31.969994+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.132766+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391880+0000","last_change":"2024-03-24T11:26:42.779061+0000","last_active":"2024-03-24T11:28:11.391880+0000","last_peered":"2024-03-24T11:28:11.391880+0000","last_clean":"2024-03-24T11:28:11.391880+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:28:11.391880+0000","last_undegraded":"2024-03-24T11:28:11.391880+0000","last_fullsized":"2024-03-24T11:28:11.391880+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.778835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.217248+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":302,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.398255+0000","last_change":"2024-03-24T11:26:41.522630+0000","last_active":"2024-03-24T11:28:11.398255+0000","last_peered":"2024-03-24T11:28:11.398255+0000","last_clean":"2024-03-24T11:28:11.398255+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:28:11.398255+0000","last_undegraded":"2024-03-24T11:28:11.398255+0000","last_fullsized":"2024-03-24T11:28:11.398255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:41.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:41.522032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.282280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":870,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:16.222277+0000","last_change":"2024-03-24T11:27:27.024036+0000","last_active":"2024-03-24T11:28:16.222277+0000","last_peered":"2024-03-24T11:28:16.222277+0000","last_clean":"2024-03-24T11:28:16.222277+0000","last_became_active":"2024-03-24T11:18:52.677967+0000","last_became_peered":"2024-03-24T11:18:52.677967+0000","last_unstale":"2024-03-24T11:28:16.222277+0000","last_undegraded":"2024-03-24T11:28:16.222277+0000","last_fullsized":"2024-03-24T11:28:16.222277+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:27:27.023866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.023866+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:56.078041+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":557,"num_read_kb":552,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.403260+0000","last_change":"2024-03-24T11:27:11.451262+0000","last_active":"2024-03-24T11:28:11.403260+0000","last_peered":"2024-03-24T11:28:11.403260+0000","last_clean":"2024-03-24T11:28:11.403260+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:28:11.403260+0000","last_undegraded":"2024-03-24T11:28:11.403260+0000","last_fullsized":"2024-03-24T11:28:11.403260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-24T11:27:11.451128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:11.451128+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:34.371691+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390165+0000","last_change":"2024-03-24T11:27:17.037155+0000","last_active":"2024-03-24T11:28:11.390165+0000","last_peered":"2024-03-24T11:28:11.390165+0000","last_clean":"2024-03-24T11:28:11.390165+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:28:11.390165+0000","last_undegraded":"2024-03-24T11:28:11.390165+0000","last_fullsized":"2024-03-24T11:28:11.390165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.037043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.037043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:18.898329+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"192'14583","reported_seq":14678,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.750426+0000","last_change":"2024-03-24T11:25:02.677146+0000","last_active":"2024-03-24T11:28:13.750426+0000","last_peered":"2024-03-24T11:28:13.750426+0000","last_clean":"2024-03-24T11:28:13.750426+0000","last_became_active":"2024-03-24T11:25:02.676840+0000","last_became_peered":"2024-03-24T11:25:02.676840+0000","last_unstale":"2024-03-24T11:28:13.750426+0000","last_undegraded":"2024-03-24T11:28:13.750426+0000","last_fullsized":"2024-03-24T11:28:13.750426+0000","mapping_epoch":150,"log_start":"183'10100","ondisk_log_start":"183'10100","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100204544,"num_objects":1529,"num_object_clones":0,"num_object_copies":4587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14583,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":435,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393487+0000","last_change":"2024-03-24T11:26:43.595952+0000","last_active":"2024-03-24T11:28:11.393487+0000","last_peered":"2024-03-24T11:28:11.393487+0000","last_clean":"2024-03-24T11:28:11.393487+0000","last_became_active":"2024-03-24T11:24:29.081667+0000","last_became_peered":"2024-03-24T11:24:29.081667+0000","last_unstale":"2024-03-24T11:28:11.393487+0000","last_undegraded":"2024-03-24T11:28:11.393487+0000","last_fullsized":"2024-03-24T11:28:11.393487+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:43.595832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:43.595832+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:13.010557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.388207+0000","last_change":"2024-03-24T11:27:21.158104+0000","last_active":"2024-03-24T11:28:11.388207+0000","last_peered":"2024-03-24T11:28:11.388207+0000","last_clean":"2024-03-24T11:28:11.388207+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:28:11.388207+0000","last_undegraded":"2024-03-24T11:28:11.388207+0000","last_fullsized":"2024-03-24T11:28:11.388207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:21.157980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:21.157980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:28.941212+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":315,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.392110+0000","last_change":"2024-03-24T11:27:25.998316+0000","last_active":"2024-03-24T11:28:11.392110+0000","last_peered":"2024-03-24T11:28:11.392110+0000","last_clean":"2024-03-24T11:28:11.392110+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:28:11.392110+0000","last_undegraded":"2024-03-24T11:28:11.392110+0000","last_fullsized":"2024-03-24T11:28:11.392110+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:25.998178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:25.998178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:29.748172+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":331,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.397364+0000","last_change":"2024-03-24T11:27:25.297250+0000","last_active":"2024-03-24T11:28:11.397364+0000","last_peered":"2024-03-24T11:28:11.397364+0000","last_clean":"2024-03-24T11:28:11.397364+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:28:11.397364+0000","last_undegraded":"2024-03-24T11:28:11.397364+0000","last_fullsized":"2024-03-24T11:28:11.397364+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-24T11:27:25.297109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:25.297109+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:35.456116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":307,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390277+0000","last_change":"2024-03-24T11:27:37.391417+0000","last_active":"2024-03-24T11:28:11.390277+0000","last_peered":"2024-03-24T11:28:11.390277+0000","last_clean":"2024-03-24T11:28:11.390277+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:28:11.390277+0000","last_undegraded":"2024-03-24T11:28:11.390277+0000","last_fullsized":"2024-03-24T11:28:11.390277+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:37.391291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:38.330738+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"192'14495","reported_seq":14653,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:14.034997+0000","last_change":"2024-03-24T11:27:20.326173+0000","last_active":"2024-03-24T11:28:14.034997+0000","last_peered":"2024-03-24T11:28:14.034997+0000","last_clean":"2024-03-24T11:28:14.034997+0000","last_became_active":"2024-03-24T11:25:02.676938+0000","last_became_peered":"2024-03-24T11:25:02.676938+0000","last_unstale":"2024-03-24T11:28:14.034997+0000","last_undegraded":"2024-03-24T11:28:14.034997+0000","last_fullsized":"2024-03-24T11:28:14.034997+0000","mapping_epoch":150,"log_start":"179'9000","ondisk_log_start":"179'9000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10606","last_scrub_stamp":"2024-03-24T11:27:20.325901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:20.325901+0000","objects_scrubbed":5503,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:46.812470+0000","scrub_duration":2923,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99155968,"num_objects":1513,"num_object_clones":0,"num_object_copies":4539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14495,"num_write_kb":512256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":435,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393618+0000","last_change":"2024-03-24T11:26:32.673765+0000","last_active":"2024-03-24T11:28:11.393618+0000","last_peered":"2024-03-24T11:28:11.393618+0000","last_clean":"2024-03-24T11:28:11.393618+0000","last_became_active":"2024-03-24T11:24:29.117106+0000","last_became_peered":"2024-03-24T11:24:29.117106+0000","last_unstale":"2024-03-24T11:28:11.393618+0000","last_undegraded":"2024-03-24T11:28:11.393618+0000","last_fullsized":"2024-03-24T11:28:11.393618+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:32.673525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:32.673525+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":288,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.396105+0000","last_change":"2024-03-24T11:27:27.391992+0000","last_active":"2024-03-24T11:28:11.396105+0000","last_peered":"2024-03-24T11:28:11.396105+0000","last_clean":"2024-03-24T11:28:11.396105+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:28:11.396105+0000","last_undegraded":"2024-03-24T11:28:11.396105+0000","last_fullsized":"2024-03-24T11:28:11.396105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:27:27.391857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.391857+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:33.529518+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":310,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393324+0000","last_change":"2024-03-24T11:27:27.949051+0000","last_active":"2024-03-24T11:28:11.393324+0000","last_peered":"2024-03-24T11:28:11.393324+0000","last_clean":"2024-03-24T11:28:11.393324+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:28:11.393324+0000","last_undegraded":"2024-03-24T11:28:11.393324+0000","last_fullsized":"2024-03-24T11:28:11.393324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:27.948926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:27.948926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:42.013446+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":318,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393174+0000","last_change":"2024-03-24T11:26:49.790900+0000","last_active":"2024-03-24T11:28:11.393174+0000","last_peered":"2024-03-24T11:28:11.393174+0000","last_clean":"2024-03-24T11:28:11.393174+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:28:11.393174+0000","last_undegraded":"2024-03-24T11:28:11.393174+0000","last_fullsized":"2024-03-24T11:28:11.393174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:26:49.790753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.790753+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":421,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.397721+0000","last_change":"2024-03-24T11:27:34.364048+0000","last_active":"2024-03-24T11:28:11.397721+0000","last_peered":"2024-03-24T11:28:11.397721+0000","last_clean":"2024-03-24T11:28:11.397721+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:28:11.397721+0000","last_undegraded":"2024-03-24T11:28:11.397721+0000","last_fullsized":"2024-03-24T11:28:11.397721+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:27:34.363907+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:34.363907+0000","last_clean_scrub_stamp":"2024-03-24T11:27:34.363907+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:53.265803+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"192'14449","reported_seq":14630,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.221788+0000","last_change":"2024-03-24T11:26:49.441112+0000","last_active":"2024-03-24T11:28:13.221788+0000","last_peered":"2024-03-24T11:28:13.221788+0000","last_clean":"2024-03-24T11:28:13.221788+0000","last_became_active":"2024-03-24T11:25:02.675745+0000","last_became_peered":"2024-03-24T11:25:02.675745+0000","last_unstale":"2024-03-24T11:28:13.221788+0000","last_undegraded":"2024-03-24T11:28:13.221788+0000","last_fullsized":"2024-03-24T11:28:13.221788+0000","mapping_epoch":150,"log_start":"180'9400","ondisk_log_start":"180'9400","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8346","last_scrub_stamp":"2024-03-24T11:26:49.440843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.440843+0000","objects_scrubbed":7686,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.854214+0000","scrub_duration":2353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910808,"num_objects":1495,"num_object_clones":0,"num_object_copies":4485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14449,"num_write_kb":510145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":429,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390608+0000","last_change":"2024-03-24T11:27:36.545219+0000","last_active":"2024-03-24T11:28:11.390608+0000","last_peered":"2024-03-24T11:28:11.390608+0000","last_clean":"2024-03-24T11:28:11.390608+0000","last_became_active":"2024-03-24T11:24:29.087468+0000","last_became_peered":"2024-03-24T11:24:29.087468+0000","last_unstale":"2024-03-24T11:28:11.390608+0000","last_undegraded":"2024-03-24T11:28:11.390608+0000","last_fullsized":"2024-03-24T11:28:11.390608+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:36.545078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:36.545078+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:37.180498+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":289,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185226+0000","last_change":"2024-03-24T11:27:28.077066+0000","last_active":"2024-03-24T11:28:15.185226+0000","last_peered":"2024-03-24T11:28:15.185226+0000","last_clean":"2024-03-24T11:28:15.185226+0000","last_became_active":"2024-03-24T11:19:00.314165+0000","last_became_peered":"2024-03-24T11:19:00.314165+0000","last_unstale":"2024-03-24T11:28:15.185226+0000","last_undegraded":"2024-03-24T11:28:15.185226+0000","last_fullsized":"2024-03-24T11:28:15.185226+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:28.076928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:28.076928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:45.991016+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":308,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.403075+0000","last_change":"2024-03-24T11:27:41.223227+0000","last_active":"2024-03-24T11:28:11.403075+0000","last_peered":"2024-03-24T11:28:11.403075+0000","last_clean":"2024-03-24T11:28:11.403075+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:28:11.403075+0000","last_undegraded":"2024-03-24T11:28:11.403075+0000","last_fullsized":"2024-03-24T11:28:11.403075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:27:41.223105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:41.223105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:56.305173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":318,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.183922+0000","last_change":"2024-03-24T11:27:15.813035+0000","last_active":"2024-03-24T11:28:15.183922+0000","last_peered":"2024-03-24T11:28:15.183922+0000","last_clean":"2024-03-24T11:28:15.183922+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:28:15.183922+0000","last_undegraded":"2024-03-24T11:28:15.183922+0000","last_fullsized":"2024-03-24T11:28:15.183922+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-03-24T11:27:15.812897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:15.812897+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:19.144490+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":395,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.392088+0000","last_change":"2024-03-24T11:25:20.232658+0000","last_active":"2024-03-24T11:28:11.392088+0000","last_peered":"2024-03-24T11:28:11.392088+0000","last_clean":"2024-03-24T11:28:11.392088+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:28:11.392088+0000","last_undegraded":"2024-03-24T11:28:11.392088+0000","last_fullsized":"2024-03-24T11:28:11.392088+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:25:20.232540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:25:20.232540+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"192'14318","reported_seq":14478,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:14.052805+0000","last_change":"2024-03-24T11:27:18.681143+0000","last_active":"2024-03-24T11:28:14.052805+0000","last_peered":"2024-03-24T11:28:14.052805+0000","last_clean":"2024-03-24T11:28:14.052805+0000","last_became_active":"2024-03-24T11:25:02.675706+0000","last_became_peered":"2024-03-24T11:25:02.675706+0000","last_unstale":"2024-03-24T11:28:14.052805+0000","last_undegraded":"2024-03-24T11:28:14.052805+0000","last_fullsized":"2024-03-24T11:28:14.052805+0000","mapping_epoch":150,"log_start":"179'8800","ondisk_log_start":"179'8800","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'10233","last_scrub_stamp":"2024-03-24T11:27:18.680945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:27:18.680945+0000","objects_scrubbed":5603,"log_size":5518,"log_dups_size":0,"ondisk_log_size":5518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:20.438526+0000","scrub_duration":2239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95682560,"num_objects":1460,"num_object_clones":0,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14318,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":431,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.185946+0000","last_change":"2024-03-24T11:26:29.924513+0000","last_active":"2024-03-24T11:28:15.185946+0000","last_peered":"2024-03-24T11:28:15.185946+0000","last_clean":"2024-03-24T11:28:15.185946+0000","last_became_active":"2024-03-24T11:24:29.082786+0000","last_became_peered":"2024-03-24T11:24:29.082786+0000","last_unstale":"2024-03-24T11:28:15.185946+0000","last_undegraded":"2024-03-24T11:28:15.185946+0000","last_fullsized":"2024-03-24T11:28:15.185946+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:26:29.924076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:26:29.924076+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":289,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184426+0000","last_change":"2024-03-24T11:27:23.794256+0000","last_active":"2024-03-24T11:28:15.184426+0000","last_peered":"2024-03-24T11:28:15.184426+0000","last_clean":"2024-03-24T11:28:15.184426+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:28:15.184426+0000","last_undegraded":"2024-03-24T11:28:15.184426+0000","last_fullsized":"2024-03-24T11:28:15.184426+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:23.794181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:49.118465+0000","last_clean_scrub_stamp":"2024-03-24T11:27:23.794181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:49.002523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.394293+0000","last_change":"2024-03-24T11:26:39.266713+0000","last_active":"2024-03-24T11:28:11.394293+0000","last_peered":"2024-03-24T11:28:11.394293+0000","last_clean":"2024-03-24T11:28:11.394293+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:28:11.394293+0000","last_undegraded":"2024-03-24T11:28:11.394293+0000","last_fullsized":"2024-03-24T11:28:11.394293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:39.265692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:26:39.265692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391537+0000","last_change":"2024-03-24T11:25:35.339628+0000","last_active":"2024-03-24T11:28:11.391537+0000","last_peered":"2024-03-24T11:28:11.391537+0000","last_clean":"2024-03-24T11:28:11.391537+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:28:11.391537+0000","last_undegraded":"2024-03-24T11:28:11.391537+0000","last_fullsized":"2024-03-24T11:28:11.391537+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-24T11:25:35.339383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:25:35.339383+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":301,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.183679+0000","last_change":"2024-03-24T11:28:11.026575+0000","last_active":"2024-03-24T11:28:15.183679+0000","last_peered":"2024-03-24T11:28:15.183679+0000","last_clean":"2024-03-24T11:28:15.183679+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:28:15.183679+0000","last_undegraded":"2024-03-24T11:28:15.183679+0000","last_fullsized":"2024-03-24T11:28:15.183679+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:11.025643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:28:11.025643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:19.482048+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"192'14510","reported_seq":14686,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.567217+0000","last_change":"2024-03-24T11:26:34.774580+0000","last_active":"2024-03-24T11:28:13.567217+0000","last_peered":"2024-03-24T11:28:13.567217+0000","last_clean":"2024-03-24T11:28:13.567217+0000","last_became_active":"2024-03-24T11:25:02.600315+0000","last_became_peered":"2024-03-24T11:25:02.600315+0000","last_unstale":"2024-03-24T11:28:13.567217+0000","last_undegraded":"2024-03-24T11:28:13.567217+0000","last_fullsized":"2024-03-24T11:28:13.567217+0000","mapping_epoch":150,"log_start":"181'9600","ondisk_log_start":"181'9600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7439","last_scrub_stamp":"2024-03-24T11:26:34.774083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:34.774083+0000","objects_scrubbed":7347,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102891520,"num_objects":1570,"num_object_clones":0,"num_object_copies":4710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14510,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":429,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393063+0000","last_change":"2024-03-24T11:25:40.019964+0000","last_active":"2024-03-24T11:28:11.393063+0000","last_peered":"2024-03-24T11:28:11.393063+0000","last_clean":"2024-03-24T11:28:11.393063+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:28:11.393063+0000","last_undegraded":"2024-03-24T11:28:11.393063+0000","last_fullsized":"2024-03-24T11:28:11.393063+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:25:40.019881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:25:40.019881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393106+0000","last_change":"2024-03-24T11:27:42.911979+0000","last_active":"2024-03-24T11:28:11.393106+0000","last_peered":"2024-03-24T11:28:11.393106+0000","last_clean":"2024-03-24T11:28:11.393106+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:28:11.393106+0000","last_undegraded":"2024-03-24T11:28:11.393106+0000","last_fullsized":"2024-03-24T11:28:11.393106+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:42.911873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:42.911873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:12.605156+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":414,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186796+0000","last_change":"2024-03-24T11:26:42.827014+0000","last_active":"2024-03-24T11:28:15.186796+0000","last_peered":"2024-03-24T11:28:15.186796+0000","last_clean":"2024-03-24T11:28:15.186796+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:28:15.186796+0000","last_undegraded":"2024-03-24T11:28:15.186796+0000","last_fullsized":"2024-03-24T11:28:15.186796+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:26:42.826873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:26:42.826873+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.391448+0000","last_change":"2024-03-24T11:25:22.291763+0000","last_active":"2024-03-24T11:28:11.391448+0000","last_peered":"2024-03-24T11:28:11.391448+0000","last_clean":"2024-03-24T11:28:11.391448+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:28:11.391448+0000","last_undegraded":"2024-03-24T11:28:11.391448+0000","last_fullsized":"2024-03-24T11:28:11.391448+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:22.291490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:25:22.291490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"89'22","reported_seq":335,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.389460+0000","last_change":"2024-03-24T11:26:46.757168+0000","last_active":"2024-03-24T11:28:11.389460+0000","last_peered":"2024-03-24T11:28:11.389460+0000","last_clean":"2024-03-24T11:28:11.389460+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:28:11.389460+0000","last_undegraded":"2024-03-24T11:28:11.389460+0000","last_fullsized":"2024-03-24T11:28:11.389460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-03-24T11:26:46.757070+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.757070+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:15.851634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"193'14562","reported_seq":14744,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:18.171599+0000","last_change":"2024-03-24T11:26:48.243638+0000","last_active":"2024-03-24T11:28:18.171599+0000","last_peered":"2024-03-24T11:28:18.171599+0000","last_clean":"2024-03-24T11:28:18.171599+0000","last_became_active":"2024-03-24T11:25:02.599329+0000","last_became_peered":"2024-03-24T11:25:02.599329+0000","last_unstale":"2024-03-24T11:28:18.171599+0000","last_undegraded":"2024-03-24T11:28:18.171599+0000","last_fullsized":"2024-03-24T11:28:18.171599+0000","mapping_epoch":150,"log_start":"184'10000","ondisk_log_start":"184'10000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'8117","last_scrub_stamp":"2024-03-24T11:26:48.243210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.243210+0000","objects_scrubbed":7656,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14562,"num_write_kb":503040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390771+0000","last_change":"2024-03-24T11:27:17.423048+0000","last_active":"2024-03-24T11:28:11.390771+0000","last_peered":"2024-03-24T11:28:11.390771+0000","last_clean":"2024-03-24T11:28:11.390771+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:28:11.390771+0000","last_undegraded":"2024-03-24T11:28:11.390771+0000","last_fullsized":"2024-03-24T11:28:11.390771+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:17.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:17.422927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:39.062528+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390072+0000","last_change":"2024-03-24T11:26:28.105930+0000","last_active":"2024-03-24T11:28:11.390072+0000","last_peered":"2024-03-24T11:28:11.390072+0000","last_clean":"2024-03-24T11:28:11.390072+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:28:11.390072+0000","last_undegraded":"2024-03-24T11:28:11.390072+0000","last_fullsized":"2024-03-24T11:28:11.390072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:28.105784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:24:43.042611+0000","last_clean_scrub_stamp":"2024-03-24T11:26:28.105784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:11.868138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":301,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.183773+0000","last_change":"2024-03-24T11:26:49.883140+0000","last_active":"2024-03-24T11:28:15.183773+0000","last_peered":"2024-03-24T11:28:15.183773+0000","last_clean":"2024-03-24T11:28:15.183773+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:28:15.183773+0000","last_undegraded":"2024-03-24T11:28:15.183773+0000","last_fullsized":"2024-03-24T11:28:15.183773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:49.883061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:49.883061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":324,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186597+0000","last_change":"2024-03-24T11:27:26.870931+0000","last_active":"2024-03-24T11:28:15.186597+0000","last_peered":"2024-03-24T11:28:15.186597+0000","last_clean":"2024-03-24T11:28:15.186597+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:28:15.186597+0000","last_undegraded":"2024-03-24T11:28:15.186597+0000","last_fullsized":"2024-03-24T11:28:15.186597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-24T11:27:26.870675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:26.870675+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:53.427130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"192'14521","reported_seq":14700,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.509411+0000","last_change":"2024-03-24T11:26:36.863675+0000","last_active":"2024-03-24T11:28:13.509411+0000","last_peered":"2024-03-24T11:28:13.509411+0000","last_clean":"2024-03-24T11:28:13.509411+0000","last_became_active":"2024-03-24T11:25:02.674932+0000","last_became_peered":"2024-03-24T11:25:02.674932+0000","last_unstale":"2024-03-24T11:28:13.509411+0000","last_undegraded":"2024-03-24T11:28:13.509411+0000","last_fullsized":"2024-03-24T11:28:13.509411+0000","mapping_epoch":150,"log_start":"181'9600","ondisk_log_start":"181'9600","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'7579","last_scrub_stamp":"2024-03-24T11:26:36.863361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:26:36.863361+0000","objects_scrubbed":7488,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99680256,"num_objects":1521,"num_object_clones":0,"num_object_copies":4563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14521,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":438,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390781+0000","last_change":"2024-03-24T11:28:08.835867+0000","last_active":"2024-03-24T11:28:11.390781+0000","last_peered":"2024-03-24T11:28:11.390781+0000","last_clean":"2024-03-24T11:28:11.390781+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:28:11.390781+0000","last_undegraded":"2024-03-24T11:28:11.390781+0000","last_fullsized":"2024-03-24T11:28:11.390781+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:28:08.835739+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:25:43.004596+0000","last_clean_scrub_stamp":"2024-03-24T11:28:08.835739+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:29.891703+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":295,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393329+0000","last_change":"2024-03-24T11:27:19.056330+0000","last_active":"2024-03-24T11:28:11.393329+0000","last_peered":"2024-03-24T11:28:11.393329+0000","last_clean":"2024-03-24T11:28:11.393329+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:28:11.393329+0000","last_undegraded":"2024-03-24T11:28:11.393329+0000","last_fullsized":"2024-03-24T11:28:11.393329+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:27:19.055992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:27:19.055992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:28.067254+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.392336+0000","last_change":"2024-03-24T11:26:48.793731+0000","last_active":"2024-03-24T11:28:11.392336+0000","last_peered":"2024-03-24T11:28:11.392336+0000","last_clean":"2024-03-24T11:28:11.392336+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:28:11.392336+0000","last_undegraded":"2024-03-24T11:28:11.392336+0000","last_fullsized":"2024-03-24T11:28:11.392336+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:26:48.793614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"66'1","reported_seq":321,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390039+0000","last_change":"2024-03-24T11:27:22.478172+0000","last_active":"2024-03-24T11:28:11.390039+0000","last_peered":"2024-03-24T11:28:11.390039+0000","last_clean":"2024-03-24T11:28:11.390039+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:28:11.390039+0000","last_undegraded":"2024-03-24T11:28:11.390039+0000","last_fullsized":"2024-03-24T11:28:11.390039+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:27:22.477947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:27:22.477947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:51.862237+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"88'25","reported_seq":365,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186504+0000","last_change":"2024-03-24T11:27:42.047337+0000","last_active":"2024-03-24T11:28:15.186504+0000","last_peered":"2024-03-24T11:28:15.186504+0000","last_clean":"2024-03-24T11:28:15.186504+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:28:15.186504+0000","last_undegraded":"2024-03-24T11:28:15.186504+0000","last_fullsized":"2024-03-24T11:28:15.186504+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'25","last_scrub_stamp":"2024-03-24T11:27:42.047201+0000","last_deep_scrub":"88'25","last_deep_scrub_stamp":"2024-03-24T11:25:49.109688+0000","last_clean_scrub_stamp":"2024-03-24T11:27:42.047201+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:54.568794+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"192'14539","reported_seq":14634,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:13.681518+0000","last_change":"2024-03-24T11:25:02.599491+0000","last_active":"2024-03-24T11:28:13.681518+0000","last_peered":"2024-03-24T11:28:13.681518+0000","last_clean":"2024-03-24T11:28:13.681518+0000","last_became_active":"2024-03-24T11:25:02.599177+0000","last_became_peered":"2024-03-24T11:25:02.599177+0000","last_unstale":"2024-03-24T11:28:13.681518+0000","last_undegraded":"2024-03-24T11:28:13.681518+0000","last_fullsized":"2024-03-24T11:28:13.681518+0000","mapping_epoch":150,"log_start":"181'10000","ondisk_log_start":"181'10000","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:01.570346+0000","last_clean_scrub_stamp":"2024-03-24T11:25:01.570346+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96927744,"num_objects":1479,"num_object_clones":0,"num_object_copies":4437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14539,"num_write_kb":512576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":579,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184509+0000","last_change":"2024-03-24T11:27:12.962801+0000","last_active":"2024-03-24T11:28:15.184509+0000","last_peered":"2024-03-24T11:28:15.184509+0000","last_clean":"2024-03-24T11:28:15.184509+0000","last_became_active":"2024-03-24T11:24:29.091858+0000","last_became_peered":"2024-03-24T11:24:29.091858+0000","last_unstale":"2024-03-24T11:28:15.184509+0000","last_undegraded":"2024-03-24T11:28:15.184509+0000","last_fullsized":"2024-03-24T11:28:15.184509+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:27:12.962703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:27:12.962703+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:31.888591+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":289,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.186807+0000","last_change":"2024-03-24T11:26:37.859404+0000","last_active":"2024-03-24T11:28:15.186807+0000","last_peered":"2024-03-24T11:28:15.186807+0000","last_clean":"2024-03-24T11:28:15.186807+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:28:15.186807+0000","last_undegraded":"2024-03-24T11:28:15.186807+0000","last_fullsized":"2024-03-24T11:28:15.186807+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:37.859286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:26:37.859286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":427,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.393814+0000","last_change":"2024-03-24T11:27:19.940085+0000","last_active":"2024-03-24T11:28:11.393814+0000","last_peered":"2024-03-24T11:28:11.393814+0000","last_clean":"2024-03-24T11:28:11.393814+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:28:11.393814+0000","last_undegraded":"2024-03-24T11:28:11.393814+0000","last_fullsized":"2024-03-24T11:28:11.393814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:27:19.939971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:27:19.939971+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:47.560194+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-24T11:28:11.390941+0000","last_change":"2024-03-24T11:26:46.366056+0000","last_active":"2024-03-24T11:28:11.390941+0000","last_peered":"2024-03-24T11:28:11.390941+0000","last_clean":"2024-03-24T11:28:11.390941+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:28:11.390941+0000","last_undegraded":"2024-03-24T11:28:11.390941+0000","last_fullsized":"2024-03-24T11:28:11.390941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:26:46.365877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:26:46.365877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":312,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-24T11:28:15.184320+0000","last_change":"2024-03-24T11:27:35.003213+0000","last_active":"2024-03-24T11:28:15.184320+0000","last_peered":"2024-03-24T11:28:15.184320+0000","last_clean":"2024-03-24T11:28:15.184320+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:28:15.184320+0000","last_undegraded":"2024-03-24T11:28:15.184320+0000","last_fullsized":"2024-03-24T11:28:15.184320+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-24T11:27:35.003081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:27:35.003081+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:40.552259+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1567621144,"num_objects":23921,"num_object_clones":0,"num_object_copies":71763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232159,"num_write_kb":8194497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4549914624,"data_stored":4549902408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80959,"ondisk_log_size":80959,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":557,"num_read_kb":552,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":451,"ondisk_log_size":451,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070581,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":625808,"kb_used_data":373896,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93119344,"statfs":{"total":95995035648,"available":95354208256,"internally_reserved":0,"allocated":382869504,"data_stored":382573652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":257948107},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364921,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781552,"kb_used_data":507308,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92963600,"statfs":{"total":95995035648,"available":95194726400,"internally_reserved":0,"allocated":519483392,"data_stored":519208257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":977624,"kb_used_data":703380,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92767528,"statfs":{"total":95995035648,"available":94993948672,"internally_reserved":0,"allocated":720261120,"data_stored":719945862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738520,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916356,"kb_used_data":596288,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92828796,"statfs":{"total":95995035648,"available":95056687104,"internally_reserved":0,"allocated":610598912,"data_stored":610302831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411470,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":873796,"kb_used_data":599676,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92871356,"statfs":{"total":95995035648,"available":95100268544,"internally_reserved":0,"allocated":614068224,"data_stored":613776568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888732,"kb_used_data":586708,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92856420,"statfs":{"total":95995035648,"available":95084974080,"internally_reserved":0,"allocated":600788992,"data_stored":600473470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986308,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653848,"kb_used_data":379728,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93091304,"statfs":{"total":95995035648,"available":95325495296,"internally_reserved":0,"allocated":388841472,"data_stored":388537832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659262,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073408,"kb_used_data":799288,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92671744,"statfs":{"total":95995035648,"available":94895865856,"internally_reserved":0,"allocated":818470912,"data_stored":818183618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":584122368,"data_stored":584122368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":703004672,"data_stored":703004672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":599527424,"data_stored":599523352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":591790080,"data_stored":591790080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":383844352,"data_stored":383844352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":806359040,"data_stored":806354968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":506920960,"data_stored":506920960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":374345728,"data_stored":374341656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:28:21.052 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T11:28:21.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-24T11:28:21.059 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:21.143 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:21.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:21.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:21.348 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:21.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:21.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:21.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:21.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:21.561 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:28:21.561 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:28:21.561 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[29608]: pgmap v377: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-24T11:28:21.561 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[29608]: 8.9 scrub starts 2024-03-24T11:28:21.561 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:28:21.561 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:28:21.561 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[29608]: 8.9 scrub ok 2024-03-24T11:28:21.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:28:21.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:28:21.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[37907]: pgmap v377: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-24T11:28:21.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[37907]: 8.9 scrub starts 2024-03-24T11:28:21.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:28:21.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:28:21.562 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:21 smithi179 ceph-mon[37907]: 8.9 scrub ok 2024-03-24T11:28:21.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:21.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:21.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:21.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:21.755 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:21.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:21.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:21 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:28:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:21 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:28:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:21 smithi181 ceph-mon[33998]: pgmap v377: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-24T11:28:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:21 smithi181 ceph-mon[33998]: 8.9 scrub starts 2024-03-24T11:28:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:21 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:28:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:21 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:28:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:21 smithi181 ceph-mon[33998]: 8.9 scrub ok 2024-03-24T11:28:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:21.914 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:21.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:22.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:22.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:22.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:22.152 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:22.190 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:22.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:22.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:22.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:22.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:22.427 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:22.462 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:22.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:22.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:22.605 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:22.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:22.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:22.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:22.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:22.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:22 smithi181 ceph-mon[33998]: 3.8 scrub starts 2024-03-24T11:28:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:22 smithi181 ceph-mon[33998]: 3.8 scrub ok 2024-03-24T11:28:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:22 smithi181 ceph-mon[33998]: 8.3 scrub starts 2024-03-24T11:28:22.913 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:22.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:22 smithi179 ceph-mon[29608]: 3.8 scrub starts 2024-03-24T11:28:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:22 smithi179 ceph-mon[29608]: 3.8 scrub ok 2024-03-24T11:28:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:22 smithi179 ceph-mon[29608]: 8.3 scrub starts 2024-03-24T11:28:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:22 smithi179 ceph-mon[37907]: 3.8 scrub starts 2024-03-24T11:28:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:22 smithi179 ceph-mon[37907]: 3.8 scrub ok 2024-03-24T11:28:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:22 smithi179 ceph-mon[37907]: 8.3 scrub starts 2024-03-24T11:28:23.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:23.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:23.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:23.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:23.181 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:23.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:23.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:23.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:23.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:23.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:23.445 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:23.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:23.493 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:28:23.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:23.591 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:23.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:23.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: pgmap v378: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:28:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:28:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:28:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: 8.3 scrub ok 2024-03-24T11:28:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/363694052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: pgmap v378: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: 8.3 scrub ok 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:28:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:28:23.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:28:23.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:28:23.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/363694052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:23.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:23.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:23.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:23.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:23.867 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:23.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:23.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: pgmap v378: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: 8.3 scrub ok 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:28:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:28:23.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/363694052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:23.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:23.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:24.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:24.104 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.143 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:24.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:24.244 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.286 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:24.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:24.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:24.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:24.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:24.563 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:24.665 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:24.703 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: 3.12 scrub starts 2024-03-24T11:28:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: 3.12 scrub ok 2024-03-24T11:28:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:28:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: 6.1b scrub starts 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: 6.1b scrub ok 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[29608]: 8.a deep-scrub starts 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: 3.12 scrub starts 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: 3.12 scrub ok 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:28:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T11:28:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: 6.1b scrub starts 2024-03-24T11:28:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: 6.1b scrub ok 2024-03-24T11:28:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:24 smithi179 ceph-mon[37907]: 8.a deep-scrub starts 2024-03-24T11:28:24.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.845 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:24.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: 3.12 scrub starts 2024-03-24T11:28:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: 3.12 scrub ok 2024-03-24T11:28:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:28:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:28:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:28:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T11:28:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:24.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: 6.1b scrub starts 2024-03-24T11:28:24.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: 6.1b scrub ok 2024-03-24T11:28:24.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:24 smithi181 ceph-mon[33998]: 8.a deep-scrub starts 2024-03-24T11:28:24.946 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:24.983 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:24.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:25.085 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:25.119 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:25.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:25.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:25.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:25.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:25.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:25.387 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:25.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:28:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:28:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:28:25.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:25.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:25.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:25.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:25.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:25.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:25.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:25.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:25.791 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:25.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:25.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:25.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:25 smithi181 ceph-mon[33998]: pgmap v380: 158 pgs: 158 active+clean; 795 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:28:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:25 smithi181 ceph-mon[33998]: 6.e scrub starts 2024-03-24T11:28:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:25 smithi181 ceph-mon[33998]: 6.e scrub ok 2024-03-24T11:28:25.937 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:25.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:25 smithi179 ceph-mon[29608]: pgmap v380: 158 pgs: 158 active+clean; 795 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:28:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:25 smithi179 ceph-mon[29608]: 6.e scrub starts 2024-03-24T11:28:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:25 smithi179 ceph-mon[29608]: 6.e scrub ok 2024-03-24T11:28:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:25 smithi179 ceph-mon[37907]: pgmap v380: 158 pgs: 158 active+clean; 795 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:28:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:25 smithi179 ceph-mon[37907]: 6.e scrub starts 2024-03-24T11:28:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:25 smithi179 ceph-mon[37907]: 6.e scrub ok 2024-03-24T11:28:26.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:26.074 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:26.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:26.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:26.209 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:26.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:26.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:26.335 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:26.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:26.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:26.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:26.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:26.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:26.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:26.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:26.699 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:26.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:26.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:26.815 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[29608]: 5.1e deep-scrub starts 2024-03-24T11:28:26.815 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[29608]: 5.1e deep-scrub ok 2024-03-24T11:28:26.815 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[29608]: 8.5 scrub starts 2024-03-24T11:28:26.815 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[29608]: 8.5 scrub ok 2024-03-24T11:28:26.815 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:28:26.815 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:28:26.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[37907]: 5.1e deep-scrub starts 2024-03-24T11:28:26.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[37907]: 5.1e deep-scrub ok 2024-03-24T11:28:26.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[37907]: 8.5 scrub starts 2024-03-24T11:28:26.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[37907]: 8.5 scrub ok 2024-03-24T11:28:26.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:28:26.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:26 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:28:26.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:26.867 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:26.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:26 smithi181 ceph-mon[33998]: 5.1e deep-scrub starts 2024-03-24T11:28:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:26 smithi181 ceph-mon[33998]: 5.1e deep-scrub ok 2024-03-24T11:28:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:26 smithi181 ceph-mon[33998]: 8.5 scrub starts 2024-03-24T11:28:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:26 smithi181 ceph-mon[33998]: 8.5 scrub ok 2024-03-24T11:28:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:26 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:28:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:26 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:28:26.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:27.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:27.102 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:27.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:27.247 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.290 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:27.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:27.391 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:27.427 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:27.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: pgmap v381: 158 pgs: 158 active+clean; 795 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: 8.1 scrub starts 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:28:27.562 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: pgmap v381: 158 pgs: 158 active+clean; 795 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: 8.1 scrub starts 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:28:27.563 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:27 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:28:27.565 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:27.667 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.710 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:27.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:27.811 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:27.862 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: pgmap v381: 158 pgs: 158 active+clean; 795 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: 8.1 scrub starts 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:28:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:27 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:28:27.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:27.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:27.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:28.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:28.134 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:28.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:28.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:28.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:28.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:28.309 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:28:28.309 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:28:28.331 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:28:28.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:28.374 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:28:28.407 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:28.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:28.412 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:28:28.451 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:28:28.485 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:28:28.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:28.526 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:28:28.540 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:28.556 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:28.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:28.574 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:28:28.611 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:28:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:28:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:28:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:28:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:28:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:28:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:28:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:28:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:28:28.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:28:28.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:28:28.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[29608]: 8.1 scrub ok 2024-03-24T11:28:28.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[29608]: 4.1d scrub starts 2024-03-24T11:28:28.639 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[29608]: 4.1d scrub ok 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[29608]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/653117124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[37907]: 8.1 scrub ok 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[37907]: 4.1d scrub starts 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[37907]: 4.1d scrub ok 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[37907]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T11:28:28.640 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/653117124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:28.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:28.688 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:28.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:28.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:28.815 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:28.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:28 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:28:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:28 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:28:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:28 smithi181 ceph-mon[33998]: 8.1 scrub ok 2024-03-24T11:28:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:28 smithi181 ceph-mon[33998]: 4.1d scrub starts 2024-03-24T11:28:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:28 smithi181 ceph-mon[33998]: 4.1d scrub ok 2024-03-24T11:28:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:28 smithi181 ceph-mon[33998]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T11:28:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/653117124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:28.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:28.950 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:28.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:29.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:29.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:29.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:29.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:29.230 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:29.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:29.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:29.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:28:29.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:29.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:29.447 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:28:29.461 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:29.536 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:29.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:29.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:29.672 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:29.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:29.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 6.1c deep-scrub starts 2024-03-24T11:28:29.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 6.1c deep-scrub ok 2024-03-24T11:28:29.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: pgmap v383: 158 pgs: 158 active+clean; 775 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 738 op/s 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 8.a deep-scrub ok 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 5.f deep-scrub starts 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 5.f deep-scrub ok 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 6.1c deep-scrub starts 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 6.1c deep-scrub ok 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: pgmap v383: 158 pgs: 158 active+clean; 775 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 738 op/s 2024-03-24T11:28:29.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:28:29.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:28:29.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 8.a deep-scrub ok 2024-03-24T11:28:29.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 5.f deep-scrub starts 2024-03-24T11:28:29.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 5.f deep-scrub ok 2024-03-24T11:28:29.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:28:29.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:29 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:28:29.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:29.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:29.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 6.1c deep-scrub starts 2024-03-24T11:28:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 6.1c deep-scrub ok 2024-03-24T11:28:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: pgmap v383: 158 pgs: 158 active+clean; 775 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 738 op/s 2024-03-24T11:28:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:28:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:28:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 8.a deep-scrub ok 2024-03-24T11:28:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 5.f deep-scrub starts 2024-03-24T11:28:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 5.f deep-scrub ok 2024-03-24T11:28:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:28:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:29 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:28:29.919 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:29.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:29.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:30.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:30.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:30.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:30.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:30.231 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:30.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:30.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:30.369 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:30.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:30.470 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:30.516 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:30.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:30.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:30.652 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:30 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:28:30.652 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:30 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:28:30.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:30 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:28:30.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:30 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:28:30.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:30.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:30.755 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:30.797 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:30.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:30.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:30.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:30 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:28:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:30 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:28:30.938 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:30.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:31.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:31.046 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:31.047 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:31.102 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:31.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:31.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:31.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:31.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:31.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:31.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:31.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:31.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:31.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:31.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:31.605 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:31.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:31.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:31.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:31.780 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:31.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:31.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:31 smithi181 ceph-mon[33998]: pgmap v384: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 481 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-24T11:28:31.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:31 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:28:31.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:31 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:28:31.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:31 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:28:31.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:31 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:28:31.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:31.920 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:31.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[29608]: pgmap v384: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 481 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-24T11:28:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:28:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:28:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:28:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:28:31.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[37907]: pgmap v384: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 481 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-24T11:28:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:28:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:28:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:28:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:31 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:28:32.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:32.065 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:32.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:32.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:32.168 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:28:32.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:32.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:32.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:32.351 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:32.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:32.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:32.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:32.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:32.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:32.616 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:32.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:32.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:32.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:32.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:32.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:32.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:32.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: 6.1e deep-scrub starts 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: 6.1e deep-scrub ok 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: 8.7 scrub starts 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:28:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:32 smithi181 ceph-mon[33998]: 8.7 scrub ok 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: 6.1e deep-scrub starts 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: 6.1e deep-scrub ok 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: 8.7 scrub starts 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:28:32.908 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[29608]: 8.7 scrub ok 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: 6.1e deep-scrub starts 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: 6.1e deep-scrub ok 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: 8.7 scrub starts 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:28:32.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:32 smithi179 ceph-mon[37907]: 8.7 scrub ok 2024-03-24T11:28:32.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:33.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:33.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:33.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:33.156 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:33.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:33.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:33.301 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:33.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:33.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:33.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:33.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:33.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:33.581 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:33.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:33.599 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:33.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:33.729 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:33.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:33.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:33.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:33.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:33.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-mon[33998]: pgmap v386: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 481 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s 2024-03-24T11:28:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-mon[33998]: 8.d scrub starts 2024-03-24T11:28:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-mon[33998]: 8.d scrub ok 2024-03-24T11:28:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:28:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:28:33.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:28:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:28:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[29608]: pgmap v386: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 481 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s 2024-03-24T11:28:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[29608]: 8.d scrub starts 2024-03-24T11:28:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[29608]: 8.d scrub ok 2024-03-24T11:28:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:28:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:28:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[37907]: pgmap v386: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 481 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s 2024-03-24T11:28:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[37907]: 8.d scrub starts 2024-03-24T11:28:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[37907]: 8.d scrub ok 2024-03-24T11:28:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:28:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:33 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:28:33.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:33.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:34.100 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:34.144 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:34.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:34.285 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:34.387 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:34.410 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:34.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:34.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:34.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:34.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:34.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:34.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:34 smithi181 ceph-mon[33998]: 4.19 scrub starts 2024-03-24T11:28:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:34 smithi181 ceph-mon[33998]: 4.19 scrub ok 2024-03-24T11:28:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:34 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:28:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:34 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:28:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:34 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:28:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:34 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:28:34.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:34.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:34.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[29608]: 4.19 scrub starts 2024-03-24T11:28:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[29608]: 4.19 scrub ok 2024-03-24T11:28:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[37907]: 4.19 scrub starts 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[37907]: 4.19 scrub ok 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:28:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:34 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:28:35.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:35.092 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:35.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:35.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:35.256 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Removed 128039 objects 2024-03-24T11:28:35.256 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Clean up completed and total clean up time :111.249 2024-03-24T11:28:35.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:35.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:35.378 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:28:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:28:35.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:35.473 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:35.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.600 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:35.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:35.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: pgmap v387: 158 pgs: 158 active+clean; 121 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: 3.2 scrub starts 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: 3.2 scrub ok 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:28:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: 8.6 scrub starts 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: 8.6 scrub ok 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4058265387' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: pgmap v387: 158 pgs: 158 active+clean; 121 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: 3.2 scrub starts 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: 3.2 scrub ok 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:28:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:28:35.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: 8.6 scrub starts 2024-03-24T11:28:35.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: 8.6 scrub ok 2024-03-24T11:28:35.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:35 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4058265387' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T11:28:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:35.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:35.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:35.850 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: pgmap v387: 158 pgs: 158 active+clean; 121 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: 3.2 scrub starts 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: 3.2 scrub ok 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:28:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: 8.6 scrub starts 2024-03-24T11:28:35.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: 8.6 scrub ok 2024-03-24T11:28:35.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:35 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4058265387' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-24T11:28:35.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:35.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:35.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:36.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:36.160 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 1 pg_num to 21 2024-03-24T11:28:36.203 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:36.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:36.329 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:36.353 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:36.455 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:36.480 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:36.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.608 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:36.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:36.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:28:36.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:28:36.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:28:36.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:28:36.702 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: 8.8 scrub starts 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: 8.8 scrub ok 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4058265387' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:28:36.703 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:28:36.704 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: 8.8 scrub starts 2024-03-24T11:28:36.704 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: 8.8 scrub ok 2024-03-24T11:28:36.704 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4058265387' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-24T11:28:36.704 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T11:28:36.704 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:36.704 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:36 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:36.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:36.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:36.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:36.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: 8.8 scrub starts 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: 8.8 scrub ok 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4058265387' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:36 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:36.960 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:36.985 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:36.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:37.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:37.111 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:37.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:37.237 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:37.338 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:37.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:37.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:37.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.616 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:37.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:37.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:37.741 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:37.843 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:37.867 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:37.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:37 smithi181 ceph-mon[33998]: pgmap v390: 158 pgs: 158 active+clean; 121 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 959 op/s 2024-03-24T11:28:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:37 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:28:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:37 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:28:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-24T11:28:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:37 smithi181 ceph-mon[33998]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T11:28:37.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[29608]: pgmap v390: 158 pgs: 158 active+clean; 121 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 959 op/s 2024-03-24T11:28:37.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[29608]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[37907]: pgmap v390: 158 pgs: 158 active+clean; 121 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 959 op/s 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-24T11:28:37.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:37 smithi179 ceph-mon[37907]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T11:28:37.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:37.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:37.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:38.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:38.119 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:38.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:38.221 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:38.244 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:38.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:38.275 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-24T11:28:38.275 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-24T11:28:38.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:38.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:38.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:38.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:38.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:38.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:38.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:38.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:38.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:38.724 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:38.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:38.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:38.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:38.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:38.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: 3.18 scrub starts 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: 3.18 scrub ok 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: 8.c scrub starts 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: 8.c scrub ok 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:38.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: 3.18 scrub starts 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: 3.18 scrub ok 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: 8.c scrub starts 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: 8.c scrub ok 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:38.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:38 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: 3.18 scrub starts 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: 3.18 scrub ok 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: 8.c scrub starts 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: 8.c scrub ok 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:38 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-24T11:28:38.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:39.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:39.105 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:39.128 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:39.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:39.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:39.355 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:39.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:39.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:39.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:39.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:39.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:39.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:39.756 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:39.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:39.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:39.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: pgmap v393: 168 pgs: 2 peering, 9 unknown, 157 active+clean; 112 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.4 scrub starts 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.4 scrub ok 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:28:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:28:39.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:28:39.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.5 deep-scrub starts 2024-03-24T11:28:39.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.5 deep-scrub ok 2024-03-24T11:28:39.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:28:39.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:28:39.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-24T11:28:39.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:39 smithi181 ceph-mon[33998]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T11:28:39.917 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: pgmap v393: 168 pgs: 2 peering, 9 unknown, 157 active+clean; 112 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-24T11:28:39.917 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.4 scrub starts 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.4 scrub ok 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.5 deep-scrub starts 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.5 deep-scrub ok 2024-03-24T11:28:39.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[29608]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: pgmap v393: 168 pgs: 2 peering, 9 unknown, 157 active+clean; 112 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.4 scrub starts 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.4 scrub ok 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:28:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:28:39.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.5 deep-scrub starts 2024-03-24T11:28:39.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.5 deep-scrub ok 2024-03-24T11:28:39.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:28:39.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:28:39.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-24T11:28:39.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:39 smithi179 ceph-mon[37907]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T11:28:39.984 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:40.008 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:40.067 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:28:40.067 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:28:40.082 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:28:40.102 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:28:40.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:40.123 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:28:40.134 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:40.143 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:28:40.164 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:28:40.191 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:28:40.192 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:28:40.192 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:28:40.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:28:40.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:28:40.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:28:40.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:28:40.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:28:40.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:28:40.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:40.261 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:40.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:40.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:40.489 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:40.551 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:40.568 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:28:40.568 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:28:40.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:40.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:28:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:28:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:28:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:28:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2386425618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T11:28:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2386425618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:28:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:40 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:28:40.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:40.804 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2386425618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:28:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:40 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:28:40.906 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:40.930 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:40.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:41.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:41.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:41.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:41.194 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:41.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:41.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:41.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:41.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:41.581 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:41.639 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:28:41.683 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:41.706 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:41.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:41.835 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 1.2 scrub starts 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 1.2 scrub ok 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: pgmap v396: 168 pgs: 2 peering, 2 unknown, 164 active+clean; 476 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 483 op/s 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:28:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:28:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:28:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T11:28:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1722274599' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:28:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:41 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:28:41.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 1.2 scrub starts 2024-03-24T11:28:41.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 1.2 scrub ok 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: pgmap v396: 168 pgs: 2 peering, 2 unknown, 164 active+clean; 476 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 483 op/s 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1722274599' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 1.2 scrub starts 2024-03-24T11:28:41.917 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 1.2 scrub ok 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: pgmap v396: 168 pgs: 2 peering, 2 unknown, 164 active+clean; 476 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 483 op/s 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1722274599' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:28:41.918 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:41 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:28:41.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:41.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:41.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:42.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:42.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:42.183 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:28:42.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:42.220 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:42.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:42.345 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:42.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.473 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:42.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:42.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:42.599 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:42.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:42.725 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:42.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.852 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:42.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:42 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:28:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:42 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:28:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:42 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:42 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:28:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:42 smithi181 ceph-mon[33998]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T11:28:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:42 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:28:42.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[37907]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:28:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:28:42.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:42.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:28:42.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[29608]: osdmap e204: 8 total, 8 up, 8 in 2024-03-24T11:28:42.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:42 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:28:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:42.980 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:42.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:43.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.106 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:43.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:43.174 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:43.207 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:43.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:43.275 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:28:43.276 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:28:43.276 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-24T11:28:43.276 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-24T11:28:43.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:43.359 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:43.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:43.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:43.586 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.611 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:43.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:43.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:43.736 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:43.838 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.862 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:43.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:43.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:28:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:28:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:28:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: pgmap v399: 168 pgs: 2 peering, 2 unknown, 164 active+clean; 476 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 444 op/s 2024-03-24T11:28:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T11:28:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:28:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:28:43.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:43.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:28:43.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:43 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:28:43.915 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:28:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: pgmap v399: 168 pgs: 2 peering, 2 unknown, 164 active+clean; 476 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 444 op/s 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: pgmap v399: 168 pgs: 2 peering, 2 unknown, 164 active+clean; 476 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 444 op/s 2024-03-24T11:28:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: osdmap e205: 8 total, 8 up, 8 in 2024-03-24T11:28:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:28:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:28:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:28:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:43 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:28:43.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:43.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:43.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:44.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:44.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:44.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:44.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:44.240 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:44.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:44.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:28:44.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:44.367 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:44.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:44.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:44.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:44.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:44.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:44.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:44.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:44.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:44.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:44.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:44.846 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:44.846 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:44 smithi179 ceph-mon[29608]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T11:28:44.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:44.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:44 smithi179 ceph-mon[37907]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T11:28:44.850 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:44.873 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:44.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:44 smithi181 ceph-mon[33998]: osdmap e206: 8 total, 8 up, 8 in 2024-03-24T11:28:44.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:45.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:45.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:28:45.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:45.114 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:28:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:28:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:28:45.128 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:45.230 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:45.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:45.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:45.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:45.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.512 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:45.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:45.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:45.617 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:45 smithi179 ceph-mon[29608]: pgmap v401: 168 pgs: 1 peering, 167 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 136 KiB/s wr, 22 op/s; 176 KiB/s, 1 objects/s recovering 2024-03-24T11:28:45.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:45 smithi179 ceph-mon[37907]: pgmap v401: 168 pgs: 1 peering, 167 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 136 KiB/s wr, 22 op/s; 176 KiB/s, 1 objects/s recovering 2024-03-24T11:28:45.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:45.695 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:45.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:45.768 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:45.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:45.893 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:45.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:45.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:45 smithi181 ceph-mon[33998]: pgmap v401: 168 pgs: 1 peering, 167 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 136 KiB/s wr, 22 op/s; 176 KiB/s, 1 objects/s recovering 2024-03-24T11:28:45.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:46.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:46.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.149 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:46.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:46.250 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:46.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:46.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:46.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:46.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:46.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:46.611 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_1' removed 2024-03-24T11:28:46.639 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:46.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:46.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:46.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3231556922' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:28:46.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:46.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:46.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3231556922' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:28:46.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3231556922' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:28:47.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:47.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.074 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:47.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:47.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:47.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:47.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:47.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:47.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:47.435 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T11:28:47.436 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T11:28:47.436 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-24T11:28:47.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-24T11:28:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.452 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:47.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:47.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:47.578 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:47.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.704 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:47.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:47.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:47.831 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:47.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:47 smithi179 ceph-mon[37907]: pgmap v403: 168 pgs: 1 peering, 167 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 109 KiB/s wr, 18 op/s; 142 KiB/s, 0 objects/s recovering 2024-03-24T11:28:47.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3231556922' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:28:47.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:47 smithi179 ceph-mon[37907]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T11:28:47.899 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:47 smithi179 ceph-mon[29608]: pgmap v403: 168 pgs: 1 peering, 167 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 109 KiB/s wr, 18 op/s; 142 KiB/s, 0 objects/s recovering 2024-03-24T11:28:47.899 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3231556922' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:28:47.899 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:47 smithi179 ceph-mon[29608]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T11:28:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:47 smithi181 ceph-mon[33998]: pgmap v403: 168 pgs: 1 peering, 167 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 109 KiB/s wr, 18 op/s; 142 KiB/s, 0 objects/s recovering 2024-03-24T11:28:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3231556922' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:28:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:47 smithi181 ceph-mon[33998]: osdmap e207: 8 total, 8 up, 8 in 2024-03-24T11:28:47.934 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:47.958 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:47.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:48.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:48.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:48.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.213 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:48.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:48.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:48.342 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:48.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.476 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:48.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:48.559 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:48.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:48.588 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:48.603 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:48.603 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:28:48.603 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:28:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:48.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:48.731 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:48.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:48.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:48.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:48.988 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:48.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:49.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:49.113 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:49.190 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:49.190 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:49.190 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:48 smithi179 ceph-mon[29608]: pgmap v405: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8 op/s; 119 KiB/s, 0 objects/s recovering 2024-03-24T11:28:49.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:49.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:49.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:48 smithi179 ceph-mon[37907]: pgmap v405: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8 op/s; 119 KiB/s, 0 objects/s recovering 2024-03-24T11:28:49.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.241 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:49.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:49.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.367 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:49.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:49.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:28:49.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:28:49.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:48 smithi181 ceph-mon[33998]: pgmap v405: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8 op/s; 119 KiB/s, 0 objects/s recovering 2024-03-24T11:28:49.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.492 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:49.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:49.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:49.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:49.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:49.724 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:49.748 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:49.850 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:49.873 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:49.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:49.999 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:49.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:50.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:50.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:50.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:50.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:50.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:50.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:50.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:50.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:50.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:50.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:50 smithi181 ceph-mon[33998]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T11:28:50.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:50 smithi179 ceph-mon[29608]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T11:28:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:50 smithi179 ceph-mon[37907]: osdmap e208: 8 total, 8 up, 8 in 2024-03-24T11:28:50.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:50.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:50.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:50.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:50.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:50.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:50.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:50.768 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:50.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:50.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:50.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:50.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:50.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:51.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:51.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:51.156 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:51.199 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:28:51.200 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:28:51.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:51.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:51.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:51.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:51 smithi181 ceph-mon[33998]: pgmap v407: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:28:51.413 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:51 smithi179 ceph-mon[29608]: pgmap v407: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:28:51.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:51 smithi179 ceph-mon[37907]: pgmap v407: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:28:51.515 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:51.538 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:51.640 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:51.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:51.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:51.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:51.939 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:51.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:52.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:52.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:52.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:52.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:52.299 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:52.324 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:52.343 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:28:52.362 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:28:52.381 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:28:52.401 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:28:52.421 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:28:52.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:52.442 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:28:52.449 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:52.463 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:28:52.484 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:28:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:28:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:28:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:28:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:28:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:28:52.485 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:28:52.486 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:28:52.486 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:28:52.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:52.578 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:52.681 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:52.705 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:52.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:52.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:52.937 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:52.961 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:52.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:53.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:53.086 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:53.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:53.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:53.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:53.269 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:28:53.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:53.346 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:53.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:53.462 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:53 smithi179 ceph-mon[29608]: pgmap v408: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 519 B/s rd, 0 op/s 2024-03-24T11:28:53.463 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1489898649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:53.463 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3279517793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:53.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:53 smithi179 ceph-mon[37907]: pgmap v408: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 519 B/s rd, 0 op/s 2024-03-24T11:28:53.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1489898649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:53.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3279517793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:53.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:53.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:53.597 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:53.602 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:53 smithi181 ceph-mon[33998]: pgmap v408: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 519 B/s rd, 0 op/s 2024-03-24T11:28:53.602 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1489898649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:28:53.602 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3279517793' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:28:53.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:28:53.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:53.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:53.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:53.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:53.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:28:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:28:53.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:53.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:53.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:53.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:54.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:54.115 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:54.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:54.122 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:54.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:54.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:54.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:54.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:54.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:54.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:54.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:54.500 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:54.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:54.509 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:54.509 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3279517793' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:28:54.509 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[29608]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T11:28:54.509 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:54.509 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[29608]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T11:28:54.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:54.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3279517793' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:28:54.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[37907]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T11:28:54.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:54.510 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:54 smithi179 ceph-mon[37907]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T11:28:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:28:54.536 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:28:54.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-24T11:28:54.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:54.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:54.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:54 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:28:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3279517793' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:28:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:54 smithi181 ceph-mon[33998]: osdmap e209: 8 total, 8 up, 8 in 2024-03-24T11:28:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:28:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:54 smithi181 ceph-mon[33998]: osdmap e210: 8 total, 8 up, 8 in 2024-03-24T11:28:54.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:54.756 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:54.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:54.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:54.882 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:54.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:54.984 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:55.008 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:55.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.135 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:55.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:55.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:28:55.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:55.258 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:28:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:28:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:28:55.263 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:55.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:55.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:55.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:55.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:55.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:55 smithi181 ceph-mon[33998]: pgmap v411: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:28:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:55.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:55 smithi179 ceph-mon[29608]: pgmap v411: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:28:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:55 smithi179 ceph-mon[37907]: pgmap v411: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:28:55.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:55.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:55.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:55.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:55.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:56.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:56.036 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:56.138 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:56.157 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:56.165 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:56.203 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:56.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:56.291 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:56.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:56.416 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:56.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:56.543 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:56.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:56.669 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:56.687 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:56.771 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:56.795 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:56.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:56.921 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:56.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:57.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:28:57.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:57.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:57.152 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.178 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:57.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:57.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:57.302 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:57.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:28:57.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:28:57.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:57.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:57.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:57 smithi179 ceph-mon[29608]: pgmap v412: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-24T11:28:57.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1299432804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T11:28:57.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:57 smithi179 ceph-mon[37907]: pgmap v412: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-24T11:28:57.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1299432804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T11:28:57.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:57 smithi181 ceph-mon[33998]: pgmap v412: 152 pgs: 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-24T11:28:57.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1299432804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T11:28:57.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:57.687 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:57.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:57.757 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_2' created 2024-03-24T11:28:57.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:57.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:57.919 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:57.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:57.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:58.048 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:58.072 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:58.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:28:58.202 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:28:58.304 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.328 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:58.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:58.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:58.438 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1299432804' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T11:28:58.438 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:58 smithi179 ceph-mon[29608]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T11:28:58.438 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1299432804' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T11:28:58.438 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:58 smithi179 ceph-mon[37907]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T11:28:58.461 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:58.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-24T11:28:58.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:58.587 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:58.689 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:58.712 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:58.815 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:58.838 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1299432804' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T11:28:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:58 smithi181 ceph-mon[33998]: osdmap e211: 8 total, 8 up, 8 in 2024-03-24T11:28:58.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:28:58.968 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:58.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:28:59.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:28:59.094 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:28:59.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:59.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:59.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:59.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:28:59.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:59.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:59.583 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:28:59.608 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:28:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:59 smithi179 ceph-mon[29608]: pgmap v414: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:28:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:59 smithi179 ceph-mon[29608]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:28:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:28:59 smithi179 ceph-mon[29608]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T11:28:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:59 smithi179 ceph-mon[37907]: pgmap v414: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:28:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:59 smithi179 ceph-mon[37907]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:28:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:28:59 smithi179 ceph-mon[37907]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T11:28:59.710 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:28:59.734 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:28:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:28:59.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:28:59.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:28:59.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:59 smithi181 ceph-mon[33998]: pgmap v414: 168 pgs: 3 creating+peering, 13 unknown, 152 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:28:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:59 smithi181 ceph-mon[33998]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:28:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:28:59 smithi181 ceph-mon[33998]: osdmap e212: 8 total, 8 up, 8 in 2024-03-24T11:28:59.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:28:59.967 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:59.967 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:28:59.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:28:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:28:59.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:00.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:00.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:00.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:00.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:00.252 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:00.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:00.353 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:00.376 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:00.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:00.478 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:00.502 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:00.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:00.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:00.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:00.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:00.692 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:00 smithi179 ceph-mon[29608]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T11:29:00.692 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:00 smithi179 ceph-mon[37907]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T11:29:00.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:00.756 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:00.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:00.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:00.882 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:00.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:00 smithi181 ceph-mon[33998]: osdmap e213: 8 total, 8 up, 8 in 2024-03-24T11:29:00.984 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:01.007 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:01.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:01.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:01.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24323.0:2366 4.1b e213/67 4:dc490aaa:::obj_delete_at_hint.0000000067:head v 213'7, mlcod=73'5)", 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:29:01.221162+0000", 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.000224935, 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025105400000000001, 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:29:01.222 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221162+0000", 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221162+0000", 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221162+0000", 2024-03-24T11:29:01.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4400000000000001e-07 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221166+0000", 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8619999999999996e-06 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221176+0000", 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8220000000000002e-06 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:01.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221181+0000", 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7910000000000004e-06 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221239+0000", 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8258999999999999e-05 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:29:01.221258+0000", 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.925e-05 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:01.225 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:01.226 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:01.226 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:29:01.226 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 1 2024-03-24T11:29:01.226 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:29:01.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:01.263 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:01.366 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:01.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:01.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:01.515 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:01.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:01.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:01.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:01 smithi179 ceph-mon[29608]: pgmap v417: 168 pgs: 2 creating+activating, 7 creating+peering, 159 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-24T11:29:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:01 smithi179 ceph-mon[37907]: pgmap v417: 168 pgs: 2 creating+activating, 7 creating+peering, 159 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-24T11:29:01.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:01.767 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:01 smithi181 ceph-mon[33998]: pgmap v417: 168 pgs: 2 creating+activating, 7 creating+peering, 159 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-24T11:29:01.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:01.893 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:01.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:01.995 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:02.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:02.120 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:02.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:02.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:02.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:02.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:02.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:02.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:02.523 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:02.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.650 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:02.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:02.751 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:02.775 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:02.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:02.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:02.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:02.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:03.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:03.036 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:03.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:03.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:03.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:03.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:03.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:03.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:03.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:03.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:03.554 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:03.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:03.682 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:03 smithi179 ceph-mon[29608]: pgmap v418: 168 pgs: 2 creating+activating, 7 creating+peering, 159 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-24T11:29:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:03 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:03 smithi179 ceph-mon[37907]: pgmap v418: 168 pgs: 2 creating+activating, 7 creating+peering, 159 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-24T11:29:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:03 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:03.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:03.809 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:03 smithi181 ceph-mon[33998]: pgmap v418: 168 pgs: 2 creating+activating, 7 creating+peering, 159 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-24T11:29:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:03 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:03.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:29:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:29:03.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:03.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:03.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:04.041 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:04.065 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:04.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.192 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:04.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:04.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:04.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:04.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:04.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:04.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:04.580 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:04.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:04.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:04.732 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:04.732 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:04 smithi179 ceph-mon[37907]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T11:29:04.732 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:04.732 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:04 smithi179 ceph-mon[29608]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T11:29:04.811 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:29:04.826 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:29:04.835 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:04 smithi181 ceph-mon[33998]: osdmap e214: 8 total, 8 up, 8 in 2024-03-24T11:29:04.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:04.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:04.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:05.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:05.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:05.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:05.221 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:05.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:05.346 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:05.428 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:29:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:29:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:29:05.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:05.473 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:05.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:05.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:05.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:05 smithi179 ceph-mon[29608]: pgmap v420: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-24T11:29:05.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:05 smithi179 ceph-mon[37907]: pgmap v420: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-24T11:29:05.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:05.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:05.854 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:29:05.858 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:05.874 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:29:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:29:05.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:05 smithi181 ceph-mon[33998]: pgmap v420: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-24T11:29:05.913 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:29:05.934 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:29:05.955 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:29:05.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:05.975 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:29:05.982 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:05.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:05.996 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:29:05.997 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:29:05.998 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:29:05.998 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:29:05.998 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:29:05.998 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:29:05.998 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:29:05.998 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:29:05.998 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:29:06.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:06.107 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:06.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:06.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:06.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:29:06.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.362 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:06.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:06.463 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:06.488 INFO:teuthology.orchestra.run.smithi179.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-24T11:29:06.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:06.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:06.614 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:06.696 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1935234489' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:29:06.697 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:06 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:29:06.697 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3956165612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:06.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1935234489' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:29:06.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:06 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:29:06.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3956165612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:06.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:06.740 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:06.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:06.866 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1935234489' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:29:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:06 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:29:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3956165612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:06.967 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:06.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:06.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:07.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:07.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:07.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:07.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:07.250 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:07.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:07.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:07.376 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:07.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:07.478 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:07.495 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:29:07.504 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:07.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:07.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:07.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:07.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[29608]: pgmap v421: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-03-24T11:29:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:29:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[29608]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T11:29:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/597992279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:29:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:29:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[37907]: pgmap v421: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-03-24T11:29:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:29:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[37907]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T11:29:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/597992279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:29:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:07 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:29:07.735 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:07.759 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:07.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:07.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:07.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:07.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:07 smithi181 ceph-mon[33998]: pgmap v421: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-03-24T11:29:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:07 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:29:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:07 smithi181 ceph-mon[33998]: osdmap e215: 8 total, 8 up, 8 in 2024-03-24T11:29:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:07 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/597992279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:29:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:07 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:29:07.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:08.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:08.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:08.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:08.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:08.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:08.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:08.272 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:08.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:08.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:08.398 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:08.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:08.419 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-24T11:29:08.443 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T11:29:08.443 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T11:29:08.494 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:hints = 1 2024-03-24T11:29:08.494 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T11:29:08.494 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object prefix: benchmark_data_smithi179_130919 2024-03-24T11:29:08.494 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:29:08.494 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-03-24T11:29:08.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:08.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:08.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:08.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:08.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:08.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:08.777 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:08 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:08.777 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:08 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:29:08.777 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:08 smithi179 ceph-mon[29608]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T11:29:08.778 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:08 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:08.778 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:08 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:29:08.778 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:08 smithi179 ceph-mon[37907]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T11:29:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:29:08.779 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:29:08.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-24T11:29:08.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:08.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:08.873 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:08.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:08 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:08 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:29:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:08 smithi181 ceph-mon[33998]: osdmap e216: 8 total, 8 up, 8 in 2024-03-24T11:29:08.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:09.009 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:09.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:09.111 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:09.153 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:09.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:09.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:09.302 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:09.304 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:09.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:09.337 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:09.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:09.449 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:09.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:09.495 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 1 16 3429 3413 213.112 213.312 0.00288976 0.00467902 2024-03-24T11:29:09.552 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:09.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:09.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:09.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[29608]: pgmap v424: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-24T11:29:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[29608]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:29:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:29:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:29:09.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[37907]: pgmap v424: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-24T11:29:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[37907]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:29:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:29:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:09 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:29:09.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:09.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:09.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:09.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:29:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:09.873 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:09.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:09 smithi181 ceph-mon[33998]: pgmap v424: 168 pgs: 168 active+clean; 607 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-24T11:29:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:09 smithi181 ceph-mon[33998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:29:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:09 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:29:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:09 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:29:09.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.008 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:10.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:10.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:10.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:10.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:10.289 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:10.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:10.431 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:10.495 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 2 16 6733 6717 209.798 206.5 0.00327999 0.00475521 2024-03-24T11:29:10.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:10.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:10.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.712 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:10.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[29608]: 1.e scrub starts 2024-03-24T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[29608]: 1.e scrub ok 2024-03-24T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[29608]: 1.14 scrub starts 2024-03-24T11:29:10.752 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[29608]: 1.14 scrub ok 2024-03-24T11:29:10.753 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:29:10.753 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:29:10.753 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[37907]: 1.e scrub starts 2024-03-24T11:29:10.753 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[37907]: 1.e scrub ok 2024-03-24T11:29:10.753 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[37907]: 1.14 scrub starts 2024-03-24T11:29:10.753 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:10 smithi179 ceph-mon[37907]: 1.14 scrub ok 2024-03-24T11:29:10.814 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:10.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:10 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:29:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:10 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:29:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:10 smithi181 ceph-mon[33998]: 1.e scrub starts 2024-03-24T11:29:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:10 smithi181 ceph-mon[33998]: 1.e scrub ok 2024-03-24T11:29:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:10 smithi181 ceph-mon[33998]: 1.14 scrub starts 2024-03-24T11:29:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:10 smithi181 ceph-mon[33998]: 1.14 scrub ok 2024-03-24T11:29:10.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:10.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:10.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:11.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:11.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:11.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:11.232 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:11.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:29:11.271 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:11.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:11.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:11.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:11.495 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 3 16 9844 9828 204.667 194.438 0.00648046 0.00487654 2024-03-24T11:29:11.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:11.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[29608]: pgmap v425: 168 pgs: 168 active+clean; 48 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 841 B/s rd, 7.8 MiB/s wr, 124 op/s 2024-03-24T11:29:11.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[29608]: 1.11 scrub starts 2024-03-24T11:29:11.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[29608]: 1.11 scrub ok 2024-03-24T11:29:11.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[29608]: 1.c scrub starts 2024-03-24T11:29:11.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[29608]: 1.c scrub ok 2024-03-24T11:29:11.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[29608]: 1.f scrub starts 2024-03-24T11:29:11.529 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[29608]: 1.f scrub ok 2024-03-24T11:29:11.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[37907]: pgmap v425: 168 pgs: 168 active+clean; 48 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 841 B/s rd, 7.8 MiB/s wr, 124 op/s 2024-03-24T11:29:11.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[37907]: 1.11 scrub starts 2024-03-24T11:29:11.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[37907]: 1.11 scrub ok 2024-03-24T11:29:11.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[37907]: 1.c scrub starts 2024-03-24T11:29:11.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[37907]: 1.c scrub ok 2024-03-24T11:29:11.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[37907]: 1.f scrub starts 2024-03-24T11:29:11.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:11 smithi179 ceph-mon[37907]: 1.f scrub ok 2024-03-24T11:29:11.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:11.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:11.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:11.688 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:11.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:11.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:11.828 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:11.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:11 smithi181 ceph-mon[33998]: pgmap v425: 168 pgs: 168 active+clean; 48 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 841 B/s rd, 7.8 MiB/s wr, 124 op/s 2024-03-24T11:29:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:11 smithi181 ceph-mon[33998]: 1.11 scrub starts 2024-03-24T11:29:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:11 smithi181 ceph-mon[33998]: 1.11 scrub ok 2024-03-24T11:29:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:11 smithi181 ceph-mon[33998]: 1.c scrub starts 2024-03-24T11:29:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:11 smithi181 ceph-mon[33998]: 1.c scrub ok 2024-03-24T11:29:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:11 smithi181 ceph-mon[33998]: 1.f scrub starts 2024-03-24T11:29:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:11 smithi181 ceph-mon[33998]: 1.f scrub ok 2024-03-24T11:29:11.929 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:11.973 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:11.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:12.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:12.120 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:12.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:12.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:12.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:12.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:12.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:12.393 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:12.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:12.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:12.496 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 4 16 12517 12501 195.266 167.062 0.0104645 0.00511362 2024-03-24T11:29:12.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:12.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:12.621 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:12.627 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:12.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:12.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:29:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:29:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 1.d scrub starts 2024-03-24T11:29:12.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 1.d scrub ok 2024-03-24T11:29:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 1.10 scrub starts 2024-03-24T11:29:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 1.10 scrub ok 2024-03-24T11:29:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 1.b scrub starts 2024-03-24T11:29:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 1.b scrub ok 2024-03-24T11:29:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:29:12.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:29:12.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 1.d scrub starts 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 1.d scrub ok 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 1.10 scrub starts 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 1.10 scrub ok 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 1.b scrub starts 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 1.b scrub ok 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:29:12.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:12 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:29:12.729 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:29:12.729 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:29:12.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:12.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:12.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 1.d scrub starts 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 1.d scrub ok 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 1.10 scrub starts 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 1.10 scrub ok 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 1.b scrub starts 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 1.b scrub ok 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:29:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:12 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:29:12.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:12.961 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:12.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:13.005 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:29:13.005 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:29:13.005 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:29:13.006 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:29:13.064 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:13.103 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:13.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:13.205 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:13.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:13.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:13.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:13.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:13.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:13.492 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:13.495 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 5 16 14844 14828 185.299 145.438 0.0032629 0.00539079 2024-03-24T11:29:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:13.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:13.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:13.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:13.672 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:13.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:13.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:13.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:13.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:13.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:13.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: pgmap v426: 168 pgs: 168 active+clean; 48 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 5.9 MiB/s wr, 94 op/s 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 1.12 scrub starts 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 1.12 scrub ok 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 1.13 scrub starts 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:29:13.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:13 smithi181 ceph-mon[33998]: 1.13 scrub ok 2024-03-24T11:29:13.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:13.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: pgmap v426: 168 pgs: 168 active+clean; 48 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 5.9 MiB/s wr, 94 op/s 2024-03-24T11:29:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:29:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:29:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 1.12 scrub starts 2024-03-24T11:29:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 1.12 scrub ok 2024-03-24T11:29:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:29:13.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 1.13 scrub starts 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[29608]: 1.13 scrub ok 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: pgmap v426: 168 pgs: 168 active+clean; 48 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 5.9 MiB/s wr, 94 op/s 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 1.12 scrub starts 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 1.12 scrub ok 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 1.13 scrub starts 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:29:13.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:13 smithi179 ceph-mon[37907]: 1.13 scrub ok 2024-03-24T11:29:14.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:14.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:14.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:14.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:14.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:14.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:14.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:14.364 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:14.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:14.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:14.495 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 6 15 16904 16889 175.885 128.812 0.00524668 0.00567935 2024-03-24T11:29:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:14.506 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:14.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:14.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:14.640 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:14.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:29:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:29:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:29:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:29:14.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:14 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:29:14.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:14.781 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:14.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:14.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:29:14.882 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:14 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:29:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:14 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:29:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:14 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:29:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:14 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:29:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:14 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:29:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:14 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:29:14.925 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:14.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:15.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:15.070 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:15.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:15.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:15.172 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:15.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:15.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:15.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:15.337 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:29:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:29:15.367 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:15.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:29:15.403 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:29:15.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-24T11:29:15.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:15.495 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 7 15 19151 19136 170.818 140.438 0.00303603 0.00584721 2024-03-24T11:29:15.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:15.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: pgmap v427: 168 pgs: 168 active+clean; 934 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:29:15.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[29608]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: pgmap v427: 168 pgs: 168 active+clean; 934 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:29:15.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:15 smithi179 ceph-mon[37907]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T11:29:15.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:15.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:15.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:15.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:15.788 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:15.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:15.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: pgmap v427: 168 pgs: 168 active+clean; 934 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 117 MiB/s wr, 1.86k op/s 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:29:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:15 smithi181 ceph-mon[33998]: osdmap e217: 8 total, 8 up, 8 in 2024-03-24T11:29:15.932 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:15.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:16.010 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:16.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:16.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:16.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:16.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:16.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:16.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:16.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:16.337 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:16.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:16.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:16.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:16.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:16.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:16.496 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 8 16 21602 21586 168.605 153.125 0.00505345 0.00592518 2024-03-24T11:29:16.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:16.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:16.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:16.710 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:16.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:16.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:16.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:16.879 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:16.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:16 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:29:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:16 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:29:16.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:16 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:29:16.982 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:17.022 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:17.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:17.124 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:17.162 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:17.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:17.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:17.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:17.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:17.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:17.453 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:17.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:17.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 9 16 23919 23903 165.943 144.812 0.00435154 0.00601983 2024-03-24T11:29:17.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:17.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:17.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:17.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:17.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:17.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:17.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:17.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:17.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:17 smithi181 ceph-mon[33998]: pgmap v429: 168 pgs: 168 active+clean; 934 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 107 MiB/s wr, 1.70k op/s 2024-03-24T11:29:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:17 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:29:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:17 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:29:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:17 smithi179 ceph-mon[29608]: pgmap v429: 168 pgs: 168 active+clean; 934 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 107 MiB/s wr, 1.70k op/s 2024-03-24T11:29:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:17 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:29:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:17 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:29:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:17 smithi179 ceph-mon[37907]: pgmap v429: 168 pgs: 168 active+clean; 934 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 107 MiB/s wr, 1.70k op/s 2024-03-24T11:29:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:17 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:29:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:17 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:29:17.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:18.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:18.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:18.088 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:18.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:18.149 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:18.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:18.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:18.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:18.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:18.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:18.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:18.443 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:18.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:18.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 10 16 26279 26263 164.098 147.5 0.00635341 0.00608685 2024-03-24T11:29:18.545 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:18.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:18.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:18.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:18.744 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:18.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:18.846 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:18.874 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:18.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:18 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:29:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:18 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:29:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:18 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:29:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:18 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:29:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:18 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:18.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:29:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:29:18.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:18.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:18.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:19.015 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:19.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:19.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:19.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:19.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:19.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:19.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:19.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 11 16 28455 28439 161.543 136 0.00374367 0.00618505 2024-03-24T11:29:19.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:19.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:19.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:19.696 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[29608]: pgmap v430: 168 pgs: 168 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-24T11:29:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:29:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:29:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[29608]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T11:29:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[37907]: pgmap v430: 168 pgs: 168 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-24T11:29:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:29:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:29:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:19 smithi179 ceph-mon[37907]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T11:29:19.799 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.834 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:19.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:19 smithi181 ceph-mon[33998]: pgmap v430: 168 pgs: 168 active+clean; 1.0 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-24T11:29:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:19 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:29:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:19 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:29:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:19 smithi181 ceph-mon[33998]: osdmap e218: 8 total, 8 up, 8 in 2024-03-24T11:29:19.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:19.974 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:19.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:20.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:20.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:20.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:20.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:20.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:20.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:20.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:20.385 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:20.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:20.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:20.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 12 16 30252 30236 157.44 112.312 0.0044986 0.0063465 2024-03-24T11:29:20.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:20.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:20.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:20.662 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:20.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:20.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:20.806 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:20.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:20.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:29:20.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:29:20.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:29:20.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:29:20.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:29:20.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:29:20.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:29:20.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:29:20.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:29:20.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:29:20.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:29:20.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:29:20.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:29:20.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:29:20.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:20 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:29:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:29:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:29:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:29:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:29:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:29:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:29:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:29:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:29:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:29:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:29:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:29:20.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:20 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:29:20.909 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:20.935 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:20.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:21.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:21.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:21.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:21.174 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:21.220 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:21.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:21.313 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:29:21.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:21.354 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:29:21.364 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:21.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:21.393 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:29:21.434 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:29:21.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:21.473 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:29:21.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 13 16 32200 32184 154.694 121.75 0.00499755 0.00645543 2024-03-24T11:29:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:21.509 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:21.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:21.512 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:29:21.551 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:29:21.586 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:29:21.589 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:29:21.589 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:29:21.589 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:29:21.589 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:29:21.590 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:29:21.590 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:29:21.590 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:29:21.590 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:29:21.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:21.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:21.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:21.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[29608]: pgmap v432: 168 pgs: 168 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 200 MiB/s wr, 3.20k op/s 2024-03-24T11:29:21.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[29608]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T11:29:21.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:29:21.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3351227458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[37907]: pgmap v432: 168 pgs: 168 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 200 MiB/s wr, 3.20k op/s 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[37907]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:29:21.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3351227458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:21.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:29:21.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:21 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:29:21.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:21.786 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:21.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:21.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:21 smithi181 ceph-mon[33998]: pgmap v432: 168 pgs: 168 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 200 MiB/s wr, 3.20k op/s 2024-03-24T11:29:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:21 smithi181 ceph-mon[33998]: osdmap e219: 8 total, 8 up, 8 in 2024-03-24T11:29:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:21 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:29:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:21 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:29:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:21 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3351227458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:21 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:29:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:21 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:29:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:21.931 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:21.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:22.032 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:22.058 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:22.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:22.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:22.197 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:22.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:22.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:22.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:22.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:22.437 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:22.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:22.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:22.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 14 16 34129 34113 152.256 120.562 0.00625802 0.00656294 2024-03-24T11:29:22.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:22.620 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:22.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:22.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:22.746 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:22.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:22.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:22.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:22.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:22 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:29:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:22 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:29:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:22 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:29:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:22 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:29:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:29:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:29:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:29:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:29:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:29:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:29:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:29:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:22 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:29:22.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.014 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:23.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:23.115 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:23.153 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:23.259 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.301 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:23.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:23.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:23.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:23.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 15 16 36021 36005 149.989 118.25 0.00960622 0.00666222 2024-03-24T11:29:23.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:23.573 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:23.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.714 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:23.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:23.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:23.848 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:23 smithi181 ceph-mon[33998]: pgmap v434: 168 pgs: 168 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 724 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T11:29:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:23 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:29:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:23 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:29:23.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:29:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:29:23.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:23 smithi179 ceph-mon[29608]: pgmap v434: 168 pgs: 168 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 724 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T11:29:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:23 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:29:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:23 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:29:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:23 smithi179 ceph-mon[37907]: pgmap v434: 168 pgs: 168 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 724 B/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-24T11:29:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:23 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:29:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:23 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:29:23.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:23.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:24.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:24.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:24.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:24.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:24.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:24.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:24.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:24.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:24.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:24.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:24.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 16 16 37697 37681 147.161 104.75 0.00622289 0.00678974 2024-03-24T11:29:24.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:24.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:24.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:24.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:24.660 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:24.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:24.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:24.799 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:24.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:24 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:29:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:24 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:29:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:24 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:29:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:24 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:29:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:24 smithi181 ceph-mon[33998]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T11:29:24.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:24.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:24.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[29608]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:29:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:29:24.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:24 smithi179 ceph-mon[37907]: osdmap e220: 8 total, 8 up, 8 in 2024-03-24T11:29:25.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:25.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:25.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:25.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:25.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:25.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:25.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:25.318 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:29:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:29:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:25.349 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:25.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:25.450 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:25.475 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:25.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:25.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 17 16 39366 39350 144.64 104.312 0.0035158 0.00689995 2024-03-24T11:29:25.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:25.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:25.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:29:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:29:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[29608]: pgmap v436: 168 pgs: 168 active+clean; 2.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 198 MiB/s wr, 3.17k op/s 2024-03-24T11:29:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[37907]: pgmap v436: 168 pgs: 168 active+clean; 2.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 198 MiB/s wr, 3.17k op/s 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:29:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:25 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:29:25.725 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:25.766 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:25.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:25.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:25.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:25.895 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:25.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:25 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:29:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:25 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:29:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:25 smithi181 ceph-mon[33998]: pgmap v436: 168 pgs: 168 active+clean; 2.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 198 MiB/s wr, 3.17k op/s 2024-03-24T11:29:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:25 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:29:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:25 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:29:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:25 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:29:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:25 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:29:25.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:26.038 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:26.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:26.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:26.179 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:26.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:26.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:26.317 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:26.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:26.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:26.453 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:26.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:26.497 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 18 16 41136 41120 142.75 110.625 0.00496448 0.00699918 2024-03-24T11:29:26.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:26.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:26.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:26.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:26.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:26.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:26.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:26.870 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:26.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:26 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:29:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:26 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:29:26.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:26 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:29:26.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:26 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:29:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:26 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:29:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:26 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:29:26.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:27.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:27.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:27.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:27.161 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:27.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:27.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:27.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:27.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:27.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:27.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:27.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 19 16 43018 43002 141.427 117.625 0.0119938 0.00706538 2024-03-24T11:29:27.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:27.594 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:27.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:27.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[29608]: 5.17 scrub starts 2024-03-24T11:29:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[29608]: 5.17 scrub ok 2024-03-24T11:29:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[29608]: pgmap v437: 168 pgs: 168 active+clean; 2.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 722 B/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-24T11:29:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:29:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:29:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:29:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:29:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[37907]: 5.17 scrub starts 2024-03-24T11:29:27.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[37907]: 5.17 scrub ok 2024-03-24T11:29:27.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[37907]: pgmap v437: 168 pgs: 168 active+clean; 2.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 722 B/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-24T11:29:27.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:29:27.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:29:27.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:29:27.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:27 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:29:27.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:27.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:27.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:27.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:27.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:27.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:27 smithi181 ceph-mon[33998]: 5.17 scrub starts 2024-03-24T11:29:27.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:27 smithi181 ceph-mon[33998]: 5.17 scrub ok 2024-03-24T11:29:27.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:27 smithi181 ceph-mon[33998]: pgmap v437: 168 pgs: 168 active+clean; 2.2 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 722 B/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-24T11:29:27.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:27 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:29:27.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:27 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:29:27.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:27 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:29:27.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:27 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:29:27.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:28.008 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:28.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:28.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:28.146 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:28.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:28.247 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:28.290 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:28.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:28.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26479 7.a 7:5f28562f:::benchmark_data_smithi179_78267_object26478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:29:28.432 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30943080700001, 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15488612299999999, 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26479 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.433 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117803+0000", 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2200000000000003e-07 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.434 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117824+0000", 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0343000000000002e-05 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117833+0000", 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7610000000000002e-06 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.435 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.117840+0000", 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0350000000000004e-06 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121526+0000", 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003685912 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121658+0000", 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013200899999999999 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.436 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.121793+0000", 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000134466 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272497+0000", 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15070414700000001 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272537+0000", 2024-03-24T11:29:28.437 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9535e-05 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272575+0000", 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8544999999999998e-05 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272634+0000", 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9206999999999997e-05 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.438 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272689+0000", 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4441000000000003e-05 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26483 7.2 7:427617cd:::benchmark_data_smithi179_78267_object26482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30856619500003, 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.160267666, 2024-03-24T11:29:28.439 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26483 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.440 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118667+0000", 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118668+0000", 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.221e-06 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118691+0000", 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2310000000000002e-05 2024-03-24T11:29:28.441 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118703+0000", 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2140999999999999e-05 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.118711+0000", 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6739999999999997e-06 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.442 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122939+0000", 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042282209999999999 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123051+0000", 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011178199999999999 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123184+0000", 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013356800000000001 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.443 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278443+0000", 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155258693 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278527+0000", 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4247999999999997e-05 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278810+0000", 2024-03-24T11:29:28.444 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028318000000000001 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278891+0000", 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1002999999999997e-05 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278935+0000", 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3625000000000003e-05 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.445 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26482 7.6 e99/92 7:67386a8b:::benchmark_data_smithi179_78267_object26481:head v 99'1627, mlcod=99'1625)", 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30516559300003, 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15002405799999999, 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.446 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122068+0000", 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122069+0000", 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.057e-06 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.447 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122090+0000", 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1392000000000001e-05 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122100+0000", 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9899999999999992e-06 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122107+0000", 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0010000000000004e-06 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.448 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122268+0000", 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016068699999999999 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122287+0000", 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9508999999999999e-05 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.449 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272051+0000", 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.149763221 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272092+0000", 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1201000000000001e-05 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.450 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26485 7.6 e99/92 7:687a6aed:::benchmark_data_smithi179_78267_object26484:head v 99'1628, mlcod=99'1625)", 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30510763199999, 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15087070799999999, 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.451 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122126+0000", 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8000000000000001e-07 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122148+0000", 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1658000000000002e-05 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.452 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122155+0000", 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8600000000000004e-06 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122160+0000", 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.288e-06 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270971+0000", 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14881098400000001 2024-03-24T11:29:28.453 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270992+0000", 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1002e-05 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272958+0000", 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019664079999999998 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.454 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272996+0000", 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8127999999999998e-05 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26488 7.a 7:57aba21a:::benchmark_data_smithi179_78267_object26487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30487841000001, 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.156446115, 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.455 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26488 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122355+0000", 2024-03-24T11:29:28.456 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122356+0000", 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.153e-06 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122473+0000", 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011658399999999999 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122488+0000", 2024-03-24T11:29:28.457 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5500999999999999e-05 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122496+0000", 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1389999999999995e-06 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123136+0000", 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00064008700000000001 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.458 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271318+0000", 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14818203499999999 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271479+0000", 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000160612 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.273137+0000", 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016574700000000001 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.459 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278568+0000", 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054316099999999999 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278633+0000", 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4759999999999997e-05 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.460 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278755+0000", 2024-03-24T11:29:28.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012220500000000001 2024-03-24T11:29:28.461 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.461 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.461 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.461 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278801+0000", 2024-03-24T11:29:28.461 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5958999999999998e-05 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26489 7.3 7:c26b0ca7:::benchmark_data_smithi179_78267_object26488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30472099899998, 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15559326900000001, 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.462 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26489 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122512+0000", 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.463 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122513+0000", 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.44e-07 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122535+0000", 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2070999999999999e-05 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.464 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122543+0000", 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.763e-06 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.122549+0000", 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0519999999999998e-06 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123004+0000", 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045522100000000001 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.465 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123123+0000", 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119145 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271099+0000", 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14797587000000001 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.275853+0000", 2024-03-24T11:29:28.466 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0047536339999999996 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.276560+0000", 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00070673899999999996 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278000+0000", 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014403090000000001 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278059+0000", 2024-03-24T11:29:28.467 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9514e-05 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278106+0000", 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6307000000000002e-05 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:26490 7.1 e99/92 7:8f3a7cc4:::benchmark_data_smithi179_78267_object26489:head v 99'1661, mlcod=99'1659)", 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:29:28.468 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30420190900003, 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14983597100000001, 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123031+0000", 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.469 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123032+0000", 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5700000000000002e-07 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123062+0000", 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.006e-05 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.470 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271241+0000", 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14817881199999999 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271247+0000", 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3219999999999998e-06 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271300+0000", 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3066000000000003e-05 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.471 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271320+0000", 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0044e-05 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272827+0000", 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0015071290000000001 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272867+0000", 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9981000000000002e-05 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.472 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:26493 7.3 7:cc4c69e2:::benchmark_data_smithi179_78267_object26492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30405329299998, 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155262276, 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 26493 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123180+0000", 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123181+0000", 2024-03-24T11:29:28.474 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9100000000000002e-07 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.123215+0000", 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3609000000000003e-05 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270946+0000", 2024-03-24T11:29:28.475 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14773181099999999 2024-03-24T11:29:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:28.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:28.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:28.477 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.270967+0000", 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0211e-05 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274532+0000", 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035658159999999999 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.478 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274642+0000", 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010941 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.274767+0000", 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012473399999999999 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.277988+0000", 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032216419999999998 2024-03-24T11:29:28.479 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278115+0000", 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012678200000000001 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278340+0000", 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000224589 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.480 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278405+0000", 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5335999999999999e-05 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.278442+0000", 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7444999999999998e-05 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24698.0:26479 7.a e99/92 ondisk, result = 0)", 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:29:28.481 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 490.30295371900002, 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14841649300000001, 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.482 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124280+0000", 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4300000000000003e-07 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.124285+0000", 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8049999999999997e-06 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271371+0000", 2024-03-24T11:29:28.483 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14708589 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.271380+0000", 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9910000000000008e-06 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272564+0000", 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0011840279999999999 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.484 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272570+0000", 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7880000000000003e-06 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:21:18.272696+0000", 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126348 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29142 8.f 8:f192749b:::benchmark_data_smithi179_105658_object29141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:29:28.485 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.761137+0000", 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 242.66609609400001, 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15871291300000001, 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29142 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.486 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761137+0000", 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761137+0000", 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761138+0000", 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.176e-06 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.487 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761160+0000", 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1665e-05 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761170+0000", 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8099999999999992e-06 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.488 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.761178+0000", 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6389999999999992e-06 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.765544+0000", 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043660870000000003 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.765820+0000", 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027657199999999999 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.489 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766369+0000", 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054847499999999998 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.770225+0000", 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003855851 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771695+0000", 2024-03-24T11:29:28.490 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001470917 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919776+0000", 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148080342 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919819+0000", 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2781000000000003e-05 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.491 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919850+0000", 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1597999999999998e-05 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29149 8.f 8:fb6565ee:::benchmark_data_smithi179_105658_object29148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.764256+0000", 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 242.662977162, 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15570266899999999, 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.492 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29149 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764256+0000", 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.493 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764256+0000", 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764257+0000", 2024-03-24T11:29:28.494 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.24e-06 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764285+0000", 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7521e-05 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764296+0000", 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1097999999999999e-05 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.495 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.764304+0000", 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3140000000000004e-06 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767982+0000", 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003677493 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768086+0000", 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010432700000000001 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.496 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768358+0000", 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00027155700000000002 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771182+0000", 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0028244810000000002 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.773685+0000", 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025023200000000002 2024-03-24T11:29:28.497 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919882+0000", 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14619728900000001 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919929+0000", 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7403e-05 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.498 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919959+0000", 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9626000000000001e-05 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29152 8.0 8:05cda426:::benchmark_data_smithi179_105658_object29151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.766296+0000", 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 242.66093701299999, 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15319068, 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.499 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29152 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766296+0000", 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766296+0000", 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.500 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766297+0000", 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1039999999999999e-06 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766318+0000", 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1009000000000002e-05 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766327+0000", 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8990000000000007e-06 2024-03-24T11:29:28.501 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766335+0000", 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1389999999999998e-06 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766935+0000", 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00060048000000000002 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.502 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767159+0000", 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022360500000000001 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767274+0000", 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000115157 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.770451+0000", 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031773249999999999 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.503 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771159+0000", 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00070825200000000004 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919372+0000", 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148212714 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919436+0000", 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3460000000000006e-05 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.504 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919487+0000", 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1536000000000003e-05 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29158 8.0 8:0bc0922d:::benchmark_data_smithi179_105658_object29157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 242.659630832, 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15200013400000001, 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.505 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29158 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.506 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767603+0000", 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3099999999999996e-07 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767625+0000", 2024-03-24T11:29:28.507 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1222000000000001e-05 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767636+0000", 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1581e-05 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767644+0000", 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0590000000000003e-06 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.508 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768740+0000", 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010955979999999999 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.769027+0000", 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000286643 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.769156+0000", 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012901500000000001 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.509 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771192+0000", 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020364939999999998 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774846+0000", 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036535500000000002 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919526+0000", 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14468013599999999 2024-03-24T11:29:28.510 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919569+0000", 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3591999999999999e-05 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919603+0000", 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.3312999999999998e-05 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.511 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29163 8.f 8:fe0437c8:::benchmark_data_smithi179_105658_object29162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.771176+0000", 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 242.656057309, 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148882706, 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29163 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.512 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771176+0000", 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771176+0000", 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771178+0000", 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.734e-06 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.513 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771204+0000", 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5853000000000001e-05 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771218+0000", 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4517e-05 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771226+0000", 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9920000000000007e-06 2024-03-24T11:29:28.514 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771722+0000", 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00049535699999999998 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771821+0000", 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9647000000000003e-05 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771942+0000", 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012074 2024-03-24T11:29:28.515 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.773994+0000", 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020524369999999998 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774005+0000", 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0368e-05 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919990+0000", 2024-03-24T11:29:28.516 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14598484 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920029+0000", 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9801000000000001e-05 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920059+0000", 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.942e-05 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.517 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29164 8.0 8:0915a6c1:::benchmark_data_smithi179_105658_object29163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.771244+0000", 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 242.65598901300001, 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.148498768, 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29164 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.518 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771244+0000", 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771244+0000", 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771245+0000", 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9100000000000004e-07 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.519 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771264+0000", 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9519000000000001e-05 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771271+0000", 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1110000000000002e-06 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771277+0000", 2024-03-24T11:29:28.520 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6289999999999998e-06 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771349+0000", 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1464999999999994e-05 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771481+0000", 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132352 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:29:28.521 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771617+0000", 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013571699999999999 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774906+0000", 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003288853 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774976+0000", 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0302999999999994e-05 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.522 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919638+0000", 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14466245899999999 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919712+0000", 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3567999999999998e-05 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919743+0000", 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1300999999999997e-05 2024-03-24T11:29:28.523 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29174 8.b 8:d5efc351:::benchmark_data_smithi179_105658_object29173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.776551+0000", 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 242.650682305, 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15217817, 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.524 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29174 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776551+0000", 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776551+0000", 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.525 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776553+0000", 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7129999999999999e-06 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776574+0000", 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.137e-05 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776634+0000", 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0146999999999999e-05 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.526 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776643+0000", 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3080000000000008e-06 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776699+0000", 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6517000000000001e-05 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776801+0000", 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101848 2024-03-24T11:29:28.527 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.776924+0000", 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012302299999999999 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778264+0000", 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0013400420000000001 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.778613+0000", 2024-03-24T11:29:28.528 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00034889699999999998 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.928631+0000", 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150018491 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.928687+0000", 2024-03-24T11:29:28.529 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5424999999999999e-05 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.928729+0000", 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2388999999999997e-05 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127665 8.f 8:ff943aea:::benchmark_data_smithi179_105658_object127664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.452238+0000", 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 164.97499524200001, 2024-03-24T11:29:28.530 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.152392888, 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127665 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452238+0000", 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.531 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452238+0000", 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452240+0000", 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.469e-06 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452261+0000", 2024-03-24T11:29:28.532 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1713999999999999e-05 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452278+0000", 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6895999999999999e-05 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452287+0000", 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5690000000000005e-06 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.533 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452364+0000", 2024-03-24T11:29:28.534 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6965000000000006e-05 2024-03-24T11:29:28.534 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.534 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452497+0000", 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013298100000000001 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.453230+0000", 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00073291399999999998 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.535 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.467082+0000", 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013852842000000001 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.467128+0000", 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5519999999999998e-05 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604429+0000", 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13730071599999999 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.536 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604578+0000", 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000149023 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604631+0000", 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3279000000000003e-05 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127666 8.f 8:f55f5ea7:::benchmark_data_smithi179_105658_object127665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.452850+0000", 2024-03-24T11:29:28.537 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 164.97438344899999, 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.156222834, 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127666 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452850+0000", 2024-03-24T11:29:28.538 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452850+0000", 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452851+0000", 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2099999999999995e-07 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.539 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452873+0000", 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2447e-05 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452889+0000", 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5744e-05 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.452897+0000", 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2060000000000007e-06 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.540 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.457799+0000", 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0049013629999999997 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.457954+0000", 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015555199999999999 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.458108+0000", 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015428999999999999 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.541 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.467140+0000", 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090315529999999995 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.467175+0000", 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.4595000000000003e-05 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.608456+0000", 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14128170300000001 2024-03-24T11:29:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.608945+0000", 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048899999999999996 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.609073+0000", 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012756000000000001 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127676 8.f 8:fcc02684:::benchmark_data_smithi179_105658_object127675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.460004+0000", 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 164.96722967700001, 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.158876238, 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127676 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.544 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.460004+0000", 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.460004+0000", 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.460005+0000", 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.531e-06 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.545 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.460028+0000", 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2671999999999999e-05 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.460040+0000", 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2277e-05 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.460048+0000", 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2430000000000005e-06 2024-03-24T11:29:28.546 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463976+0000", 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039278580000000002 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.464097+0000", 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120445 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.464232+0000", 2024-03-24T11:29:28.547 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013548400000000001 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.467446+0000", 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032135140000000002 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.467757+0000", 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000310916 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618784+0000", 2024-03-24T11:29:28.548 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15102700099999999 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618838+0000", 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4830999999999997e-05 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618880+0000", 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1465999999999999e-05 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.549 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:189828 8.b 8:d4aa58d2:::benchmark_data_smithi179_105658_object61786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:27:37.950924+0000", 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 110.4763093, 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17032187500000001, 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 189828 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.550 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:37.950924+0000", 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:37.950924+0000", 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:37.950925+0000", 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6599999999999994e-07 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:29:28.551 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:37.950929+0000", 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0010000000000003e-06 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.004115+0000", 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.053186300999999998 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.004134+0000", 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8389999999999998e-05 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.552 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.004201+0000", 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7712000000000005e-05 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.004325+0000", 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123206 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.004445+0000", 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120797 2024-03-24T11:29:28.553 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.012777+0000", 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0083315479999999994 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.012877+0000", 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9734000000000004e-05 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.121132+0000", 2024-03-24T11:29:28.554 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10825522999999999 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.121192+0000", 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0361e-05 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:27:38.121246+0000", 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3628999999999998e-05 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.555 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:29:28.556 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:29:28.556 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:29:28.556 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:29:28.498303+0000 min lat: 0.00143742 max lat: 0.120414 avg lat: 0.00715833 2024-03-24T11:29:28.556 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:29:28.556 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 20 16 44692 44676 139.587 104.625 0.0110079 0.00715833 2024-03-24T11:29:28.556 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:29:28.556 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:29:28.556 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:29:28.556 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:29:28.557 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:29:28.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:28.612 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:28.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[29608]: 3.b scrub starts 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[29608]: 3.b scrub ok 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[37907]: 3.b scrub starts 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[37907]: 3.b scrub ok 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:29:28.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:28 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:29:28.713 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:28.755 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:28.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:28.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:28.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:28.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:28 smithi181 ceph-mon[33998]: 3.b scrub starts 2024-03-24T11:29:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:28 smithi181 ceph-mon[33998]: 3.b scrub ok 2024-03-24T11:29:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:28 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:29:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:28 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:29:28.995 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:29.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:29.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:29:29.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:29.167 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:29.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:29.270 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:29.318 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:29.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:29.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:29.455 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:29.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:29.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 21 15 46380 46365 137.966 105.562 0.00663143 0.00724322 2024-03-24T11:29:29.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:29.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:29:29.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:29.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:29.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:29.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:29.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:29.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:29.877 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:29.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:29.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:29:29.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[29608]: 3.7 scrub starts 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[29608]: 3.7 scrub ok 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[29608]: pgmap v438: 168 pgs: 168 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[37907]: 3.7 scrub starts 2024-03-24T11:29:29.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[37907]: 3.7 scrub ok 2024-03-24T11:29:29.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[37907]: pgmap v438: 168 pgs: 168 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T11:29:29.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:29:29.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:29 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:29:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:29 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:29:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:29 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:29:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:29 smithi181 ceph-mon[33998]: 3.7 scrub starts 2024-03-24T11:29:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:29 smithi181 ceph-mon[33998]: 3.7 scrub ok 2024-03-24T11:29:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:29 smithi181 ceph-mon[33998]: pgmap v438: 168 pgs: 168 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T11:29:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:29 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:29:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:29 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:29:29.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:30.028 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:30.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:30.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:30.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:30.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:30.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:30.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:30.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:30.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:30.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:30.448 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:30.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:30.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 22 15 48024 48009 136.365 102.75 0.00718009 0.00732937 2024-03-24T11:29:30.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:30.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:30.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:30.623 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:29:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:29:30.692 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:29:30.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-24T11:29:30.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:30.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:30.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:30.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:30.901 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:30.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:30 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:29:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:30 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:29:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:30 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3901122179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:30 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:30 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:29:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:29:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:29:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3901122179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:29:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:29:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:29:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3901122179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:30 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:29:31.003 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:31.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:31.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:31.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:31.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:31.176 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:31.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:31.278 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:31.301 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:31.329 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:31.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:31.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:31.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:31.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:31.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 23 16 49620 49604 134.77 99.6875 0.00945593 0.00741498 2024-03-24T11:29:31.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:31.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:31.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:31.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:31.761 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:31.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:31.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: 4.10 scrub starts 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: 4.10 scrub ok 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: pgmap v439: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[29608]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T11:29:31.879 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: 4.10 scrub starts 2024-03-24T11:29:31.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: 4.10 scrub ok 2024-03-24T11:29:31.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: pgmap v439: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-24T11:29:31.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:29:31.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:29:31.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:29:31.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:29:31.880 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:31 smithi179 ceph-mon[37907]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T11:29:31.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: 4.10 scrub starts 2024-03-24T11:29:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: 4.10 scrub ok 2024-03-24T11:29:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: pgmap v439: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-24T11:29:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:29:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:29:31.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:29:31.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:29:31.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:31 smithi181 ceph-mon[33998]: osdmap e221: 8 total, 8 up, 8 in 2024-03-24T11:29:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:31.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:32.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:32.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:32.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:32.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:32.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:32.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:32.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:32.290 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:50880 9.c e221/211 9:3aa0d488:::benchmark_data_smithi179_130919_object50879:head v 221'3105, mlcod=221'3102)", 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:32.288681+0000", 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0011957199999999999, 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001227828, 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288681+0000", 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:32.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288681+0000", 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288688+0000", 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5339999999999996e-06 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288744+0000", 2024-03-24T11:29:32.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5804000000000003e-05 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288754+0000", 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0593e-05 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288762+0000", 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0700000000000007e-06 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288826+0000", 2024-03-24T11:29:32.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3658000000000006e-05 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:32.288845+0000", 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8873e-05 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2024-03-24T11:29:32.294 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:29:32.314 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:32.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:32.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:32.453 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:32.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:32.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 24 16 51209 51193 133.293 99.3125 0.00494029 0.00749725 2024-03-24T11:29:32.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:32.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:32.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:32.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:32.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:32.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:32.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:32.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:32 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:29:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:32 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:29:32.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:32 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:29:32.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:32 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:29:32.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:32 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:29:32.907 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:32 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:29:32.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:33.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:33.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:33.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:33.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:33.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:33.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:33.306 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:33.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:33.351 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:33.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:29:33.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:29:33.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:33.460 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:33.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:33.477 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:29:33.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 25 16 52657 52641 131.581 90.5 0.00855164 0.00759363 2024-03-24T11:29:33.515 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:29:33.559 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:29:33.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:33.595 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:29:33.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:33.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:33.633 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:29:33.677 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:29:33.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:33.721 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:29:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:33.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:33.766 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:29:33.767 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: 4.14 scrub starts 2024-03-24T11:29:33.767 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: 4.14 scrub ok 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: pgmap v441: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 633 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/762781300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:33.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: 4.14 scrub starts 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: 4.14 scrub ok 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: pgmap v441: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 633 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/762781300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:33.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:33 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:29:33.770 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:29:33.779 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:33.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:33.902 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: 4.14 scrub starts 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: 4.14 scrub ok 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: pgmap v441: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 633 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/762781300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:29:33.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:29:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:29:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:33.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:34.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:34.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:34.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:34.142 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:34.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:34.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:34.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:34.318 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:34.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:34.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:34.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:34.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:34.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 26 16 54119 54103 130.034 91.375 0.0121087 0.00768516 2024-03-24T11:29:34.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:34.591 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:34.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:34.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:29:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:29:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: 1.8 scrub starts 2024-03-24T11:29:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: 1.8 scrub ok 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: 1.8 scrub starts 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: 1.8 scrub ok 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:29:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T11:29:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:29:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:34 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:29:34.730 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:34.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:34.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:34.870 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:34.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: 1.8 scrub starts 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: 1.8 scrub ok 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: osdmap e222: 8 total, 8 up, 8 in 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:29:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:34 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:29:34.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:35.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:35.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:35.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:35.149 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:35.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:35.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:29:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:29:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:29:35.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:35.289 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:35.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:35.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:35.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:35.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:35.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 27 16 55727 55711 128.94 100.5 0.0235767 0.00775046 2024-03-24T11:29:35.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:35.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:35.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:35.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:35.713 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:35.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:35.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:35.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:35.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:35.869 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:29:35.869 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:29:35.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:35 smithi179 ceph-mon[29608]: pgmap v443: 168 pgs: 168 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-24T11:29:35.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:35 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:29:35.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:35 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:29:35.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:35 smithi179 ceph-mon[37907]: pgmap v443: 168 pgs: 168 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-24T11:29:35.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:35 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:29:35.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:35 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:29:35.884 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:35 smithi181 ceph-mon[33998]: pgmap v443: 168 pgs: 168 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-03-24T11:29:35.884 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:35 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:29:35.884 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:35 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:29:35.938 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:35.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:36.026 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:29:36.026 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:29:36.027 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:29:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:36.028 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:36.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:36.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:36.168 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:36.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:36.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:36.309 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:36.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:36.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:36.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:36.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:36.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 28 15 57198 57183 127.62 92 0.0119107 0.0078313 2024-03-24T11:29:36.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:36.583 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:36.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:36.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:36.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:36.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:36.864 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:36.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:36.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:29:36.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:37.006 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:37.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:37.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:37.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:37.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:37.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:37.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:37.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:37.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:37.329 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:37.355 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:29:37.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:37.440 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:37.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:37.498 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 29 16 58705 58689 126.465 94.125 0.00718684 0.00790237 2024-03-24T11:29:37.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:37.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:37.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:37.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:37.712 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:37.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:37.733 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:37 smithi181 ceph-mon[33998]: pgmap v444: 168 pgs: 168 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-24T11:29:37.733 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:37 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:29:37.733 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:37 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:29:37.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:37 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:29:37.734 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:37 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:29:37.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:37.885 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:37.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[29608]: pgmap v444: 168 pgs: 168 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[37907]: pgmap v444: 168 pgs: 168 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:29:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:37 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:29:37.987 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:38.030 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:38.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:38.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:38.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:38.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:38.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:38.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:38.499 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 30 15 60226 60211 125.42 95.125 0.00701305 0.00796808 2024-03-24T11:29:38.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:29:38.549 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:29:38.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-24T11:29:38.581 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:38.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:38.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:38.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:38 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:29:38.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:38 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:29:38.797 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:38 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:29:38.797 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:38 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:29:38.805 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:38.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:38.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:38.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:38 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:29:38.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:38 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:29:38.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:38.990 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:38.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:39.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:39.146 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:39.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:39.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:39.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:39.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:39.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:39.375 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:29:39.376 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:29:39.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:39.430 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:39.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:39.499 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 31 16 61541 61525 124.024 82.125 0.0127179 0.0080577 2024-03-24T11:29:39.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:39.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:39.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:39.666 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:29:39.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:39.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:39.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:39.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:39.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:39.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:39.939 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: pgmap v445: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:29:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/913896162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: pgmap v445: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/913896162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:29:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:39 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:29:39.975 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:39.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:40.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: pgmap v445: 168 pgs: 168 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:29:40.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:29:40.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:29:40.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:29:40.041 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:29:40.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:29:40.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:29:40.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/913896162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:29:40.042 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:39 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:29:40.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:40.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:29:40.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:40.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:40.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:40.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:40.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:40.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:40.394 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:40.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:40.430 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:40.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:40.499 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 32 16 62944 62928 122.886 87.6875 0.00935471 0.00813285 2024-03-24T11:29:40.505 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:40.541 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:40.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:40.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:40.685 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:40.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:40.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:40.822 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:40.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: 6.c scrub starts 2024-03-24T11:29:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: 6.c scrub ok 2024-03-24T11:29:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:29:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:29:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:29:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T11:29:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:29:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:40 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:29:40.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:40.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: 6.c scrub starts 2024-03-24T11:29:40.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: 6.c scrub ok 2024-03-24T11:29:40.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: 6.c scrub starts 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: 6.c scrub ok 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:40.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:29:40.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:29:40.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: osdmap e223: 8 total, 8 up, 8 in 2024-03-24T11:29:40.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:29:40.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:40 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:29:40.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:40.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:41.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:41.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:41.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:41.205 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:41.250 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:41.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:41.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:41.385 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:41.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:41.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:41.499 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 33 16 64430 64414 121.976 92.875 0.0135187 0.00819339 2024-03-24T11:29:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:41.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:41.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:41.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:41.650 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:41.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:41.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:41 smithi181 ceph-mon[33998]: pgmap v447: 168 pgs: 168 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-24T11:29:41.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:41 smithi181 ceph-mon[33998]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T11:29:41.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:41.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:41.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:41.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:41.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:41.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:41.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:41.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:41 smithi179 ceph-mon[29608]: pgmap v447: 168 pgs: 168 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-24T11:29:41.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:41 smithi179 ceph-mon[29608]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T11:29:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:41 smithi179 ceph-mon[37907]: pgmap v447: 168 pgs: 168 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-24T11:29:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:41 smithi179 ceph-mon[37907]: osdmap e224: 8 total, 8 up, 8 in 2024-03-24T11:29:42.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:42.073 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:42.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:42.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:42.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:42.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:42.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:42.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:42.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:42.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:42.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:42.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:42.499 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 34 16 65751 65735 120.817 82.5625 0.0108737 0.00827176 2024-03-24T11:29:42.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:42.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:42.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:42.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:42.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:42.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:42.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:42.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:42.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:42 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:29:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:42 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:29:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:42 smithi179 ceph-mon[29608]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T11:29:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:42 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:29:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:42 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:29:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:42 smithi179 ceph-mon[37907]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T11:29:42.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:43.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:29:43.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:43.127 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:42 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:29:43.127 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:42 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:29:43.127 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:42 smithi181 ceph-mon[33998]: osdmap e225: 8 total, 8 up, 8 in 2024-03-24T11:29:43.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:43.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:43.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:43.293 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:43.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:43.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:43.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 35 16 66890 66874 119.398 71.1875 0.0101439 0.00836966 2024-03-24T11:29:43.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:43.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:43.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:43.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:43.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:43.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:43 smithi181 ceph-mon[33998]: pgmap v450: 168 pgs: 168 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:29:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:43 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:43.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:29:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:29:43.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:43.957 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:43.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:43 smithi179 ceph-mon[29608]: pgmap v450: 168 pgs: 168 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:29:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:43 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:43.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:43 smithi179 ceph-mon[37907]: pgmap v450: 168 pgs: 168 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:29:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:43 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:29:44.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:44.121 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:44.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:44.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:44.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:44.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:44.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:44.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:44.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:44.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 36 16 68043 68027 118.083 72.0625 0.00738944 0.00846339 2024-03-24T11:29:44.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:44.549 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:44.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:44.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:44.671 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:29:44.671 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:29:44.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:44.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:44.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:44.851 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:44.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T11:29:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:44 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:44.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T11:29:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: osdmap e226: 8 total, 8 up, 8 in 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:29:44.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:44 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:45.002 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:45.104 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.143 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:45.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:45.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:45.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:45.302 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:29:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:29:45.382 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.415 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:45.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:45.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 37 16 69271 69255 116.966 76.75 0.00543423 0.00854479 2024-03-24T11:29:45.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:45.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:45.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:45.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:45.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:45.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:29:45.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:45.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:45.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[29608]: pgmap v452: 168 pgs: 168 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T11:29:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi181 to 3248M 2024-03-24T11:29:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:29:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:29:45.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[37907]: pgmap v452: 168 pgs: 168 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T11:29:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi181 to 3248M 2024-03-24T11:29:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:29:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:29:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:45 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:45.967 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:45.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:46.066 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:46.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:46.108 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:46.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:45 smithi181 ceph-mon[33998]: pgmap v452: 168 pgs: 168 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T11:29:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:45 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi181 to 3248M 2024-03-24T11:29:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:45 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:29:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:45 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:29:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:45 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:46.211 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:46.256 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:46.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:46.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:46.400 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:46.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:46.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 38 16 70398 70382 115.742 70.4375 0.0109884 0.00863506 2024-03-24T11:29:46.501 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:46.543 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:46.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:46.578 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:29:46.579 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:29:46.580 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:29:46.580 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:29:46.580 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:29:46.580 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:29:46.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:46.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:46.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:46.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:46.808 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:46.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1322800365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:46.909 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:46.937 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:46.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1322800365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:46.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1322800365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:47.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:47.079 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:47.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:47.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:47.216 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:47.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:47.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:47.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:47.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:47.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:47.484 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:47.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:47.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 39 16 71529 71513 114.586 70.6875 0.0372426 0.0087185 2024-03-24T11:29:47.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:47.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:47.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:47.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:47.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:47.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:47.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:47.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:47.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:47.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:47 smithi181 ceph-mon[33998]: pgmap v453: 168 pgs: 168 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-03-24T11:29:47.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:47 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:29:47.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:47 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:29:48.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:48.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:48.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:48.148 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:29:48.148 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:29:48.148 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:29:48.148 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:29:48.148 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:29:48.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:48.184 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:48.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:47 smithi179 ceph-mon[29608]: pgmap v453: 168 pgs: 168 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-03-24T11:29:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:47 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:29:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:47 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:29:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:47 smithi179 ceph-mon[37907]: pgmap v453: 168 pgs: 168 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-03-24T11:29:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:47 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:29:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:47 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:29:48.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:48.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:48.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:48.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:48.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:48.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:48.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:29:48.500752+0000 min lat: 0.00143742 max lat: 0.133698 avg lat: 0.00881172 2024-03-24T11:29:48.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:29:48.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 40 16 72603 72587 113.4 67.125 0.00395858 0.00881172 2024-03-24T11:29:48.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:48.596 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:48.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:48.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:48.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:48.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:48.838 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:48.879 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:48.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:48 smithi181 ceph-mon[33998]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T11:29:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:48.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:49.014 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:49.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:49.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:29:49.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:48 smithi179 ceph-mon[29608]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T11:29:49.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:49.112 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:49.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:48 smithi179 ceph-mon[37907]: osdmap e227: 8 total, 8 up, 8 in 2024-03-24T11:29:49.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:29:49.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:29:49.117 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:49.156 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:49.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:49.259 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:49.293 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:49.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:49.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:49.426 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:49.432 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:49.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:49.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 41 16 73791 73775 112.445 74.25 0.0208767 0.00888773 2024-03-24T11:29:49.534 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:49.584 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:49.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:49.670 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:49.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:49.713 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:29:49.713 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:29:49.713 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:29:49.713 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:29:49.713 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:29:49.713 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:29:49.714 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:29:49.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:29:49.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:29:49.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:29:49.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:29:49.714 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:29:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:49.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:49.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:49.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:49 smithi181 ceph-mon[33998]: pgmap v455: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:29:49.814 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:49 smithi181 ceph-mon[33998]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T11:29:49.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:49.876 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:49.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:49.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:50.014 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:50.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:50.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:50.153 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:50.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:50.189 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:49 smithi179 ceph-mon[29608]: pgmap v455: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:29:50.189 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:49 smithi179 ceph-mon[29608]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T11:29:50.189 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:49 smithi179 ceph-mon[37907]: pgmap v455: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:29:50.189 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:49 smithi179 ceph-mon[37907]: osdmap e228: 8 total, 8 up, 8 in 2024-03-24T11:29:50.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:50.292 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:50.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:50.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:50.434 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:50.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:50.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 42 16 75114 75098 111.736 82.6875 0.0124723 0.00894427 2024-03-24T11:29:50.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:50.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:50.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:50.677 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:50.721 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:50.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:50.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:50.864 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:50.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:50.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:51.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:51.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:51.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:29:51.148 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:29:51.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-24T11:29:51.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:51.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:51.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:51.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:51.500 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 43 16 76311 76295 110.878 74.8125 0.00624594 0.00901395 2024-03-24T11:29:51.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.546 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:51.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:51.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:51.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:51.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:51.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:51.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:51.952 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:51.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:52.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:52.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:52.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:52.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:51 smithi181 ceph-mon[33998]: pgmap v457: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 841 B/s rd, 59 MiB/s wr, 938 op/s 2024-03-24T11:29:52.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:52.196 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:51 smithi179 ceph-mon[29608]: pgmap v457: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 841 B/s rd, 59 MiB/s wr, 938 op/s 2024-03-24T11:29:52.200 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:51 smithi179 ceph-mon[37907]: pgmap v457: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 841 B/s rd, 59 MiB/s wr, 938 op/s 2024-03-24T11:29:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:52.217 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:52.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:52.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:29:52.270 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:52.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:52.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:52.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:52.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:52.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 44 16 77438 77422 109.956 70.4375 0.0118765 0.00908915 2024-03-24T11:29:52.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:52.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:52.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:52.657 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:52.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:52.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:52.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:52.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:52.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:52.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:52.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:53.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:53.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:53.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:53.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:53.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:53.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:53.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:53.292 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:53.333 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:53.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:53.408 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:29:53.408 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:29:53.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:29:53.408 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:29:53.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26498 7.0 7:0d5c2bb1:::benchmark_data_smithi179_78267_object26497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 515.27763872100002, 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14803461000000001, 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26498 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125527+0000", 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125528+0000", 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.584e-06 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125569+0000", 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0945000000000002e-05 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125586+0000", 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7198999999999999e-05 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125596+0000", 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4919999999999999e-06 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125670+0000", 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4246999999999996e-05 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125813+0000", 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014268000000000001 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126002+0000", 2024-03-24T11:29:53.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000189626 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126981+0000", 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00097871299999999998 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127606+0000", 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062507399999999999 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273447+0000", 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14584118800000001 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273512+0000", 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4628999999999994e-05 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.273561+0000", 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9233000000000001e-05 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.414 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126626 7.0 7:0b9b7f0f:::benchmark_data_smithi179_78267_object7432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 428.82694221100002, 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10890327499999999, 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126626 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576223+0000", 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576224+0000", 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.172e-06 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576229+0000", 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5279999999999997e-06 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576240+0000", 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0963e-05 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576248+0000", 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6589999999999996e-06 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576302+0000", 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3278e-05 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576422+0000", 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119988 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576496+0000", 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4645000000000001e-05 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.579163+0000", 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002667123 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580732+0000", 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001568124 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685008+0000", 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10427672 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685071+0000", 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2853000000000003e-05 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685126+0000", 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5222000000000001e-05 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126640 7.0 7:05a202e2:::benchmark_data_smithi179_78267_object7446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:29:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 428.811993986, 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097235629000000004, 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126640 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591171+0000", 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591173+0000", 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.327e-06 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591177+0000", 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0589999999999996e-06 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591188+0000", 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0954e-05 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591194+0000", 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3400000000000003e-06 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591252+0000", 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7930999999999998e-05 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591368+0000", 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115586 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591497+0000", 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129834 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.593983+0000", 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024858359999999999 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.594318+0000", 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033431900000000002 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688305+0000", 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093987494000000005 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688361+0000", 2024-03-24T11:29:53.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6203999999999999e-05 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.688407+0000", 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5745000000000003e-05 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:154480 7.0 7:09166c18:::benchmark_data_smithi179_78267_object35286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 398.303490205, 2024-03-24T11:29:53.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099165692999999999, 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 154480 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099675+0000", 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099677+0000", 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.575e-06 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099684+0000", 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6120000000000001e-06 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099697+0000", 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2541000000000001e-05 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099704+0000", 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5499999999999997e-06 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.099976+0000", 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000271122 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.100084+0000", 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010849900000000001 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.100211+0000", 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012710099999999999 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.103105+0000", 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028942939999999999 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106238+0000", 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003132953 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198758+0000", 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092520054000000004 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198797+0000", 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8912000000000003e-05 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.198841+0000", 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3479999999999997e-05 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:216710 7.0 7:0ea60a8d:::benchmark_data_smithi179_78267_object97516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:19.012852+0000", 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 334.390313741, 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.129447284, 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.432 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 216710 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012852+0000", 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012852+0000", 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012853+0000", 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.114e-06 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012857+0000", 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6360000000000002e-06 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012868+0000", 2024-03-24T11:29:53.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1141000000000001e-05 2024-03-24T11:29:53.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012875+0000", 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2870000000000001e-06 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.012923+0000", 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8678999999999999e-05 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013052+0000", 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012856200000000001 2024-03-24T11:29:53.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013130+0000", 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7836999999999994e-05 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017366+0000", 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004235684 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017565+0000", 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019948200000000001 2024-03-24T11:29:53.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142194+0000", 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.124628823 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142255+0000", 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0783999999999999e-05 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142299+0000", 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4255000000000002e-05 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:216711 7.0 7:01cdc540:::benchmark_data_smithi179_78267_object97517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 334.39014663900002, 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12939452500000001, 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.439 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 216711 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013019+0000", 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.37e-07 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013023+0000", 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0489999999999997e-06 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013034+0000", 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0847e-05 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013040+0000", 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1739999999999997e-06 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.015341+0000", 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023009079999999999 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.015431+0000", 2024-03-24T11:29:53.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9692999999999997e-05 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.015511+0000", 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9919000000000005e-05 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017510+0000", 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001999163 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.017684+0000", 2024-03-24T11:29:53.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017392800000000001 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142343+0000", 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12465865199999999 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142377+0000", 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3954000000000001e-05 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.142413+0000", 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6501000000000002e-05 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:15737 8.8 8:1331dead:::benchmark_data_smithi179_105658_object15736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 273.90076474799997, 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.108844888, 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 15737 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:29:53.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502401+0000", 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2699999999999999e-07 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502439+0000", 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8068000000000002e-05 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502447+0000", 2024-03-24T11:29:53.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2619999999999998e-06 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502454+0000", 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2859999999999996e-06 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502522+0000", 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8664000000000005e-05 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502628+0000", 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105179 2024-03-24T11:29:53.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.502779+0000", 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151512 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.504075+0000", 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00129641 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.504167+0000", 2024-03-24T11:29:53.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1201999999999999e-05 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.611160+0000", 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10699328299999999 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.611208+0000", 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7743e-05 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.611245+0000", 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7752e-05 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:164399 8.8 8:11f66a3a:::benchmark_data_smithi179_105658_object36357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.307947+0000", 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 158.09521854799999, 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12982538800000001, 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164399 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.307947+0000", 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.307947+0000", 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.307948+0000", 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6700000000000002e-07 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.307951+0000", 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4130000000000002e-06 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.307963+0000", 2024-03-24T11:29:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1861e-05 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.307970+0000", 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2259999999999999e-06 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.308129+0000", 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015910800000000001 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.308241+0000", 2024-03-24T11:29:53.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111276 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.308370+0000", 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012987099999999999 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314268+0000", 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058971830000000003 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314296+0000", 2024-03-24T11:29:53.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8646999999999999e-05 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.437638+0000", 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.123342247 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.437689+0000", 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0637999999999999e-05 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.437772+0000", 2024-03-24T11:29:53.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3051000000000002e-05 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:189854 8.8 8:1f219966:::benchmark_data_smithi179_105658_object61812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:37.986210+0000", 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 135.41695482399999, 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.141550496, 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189854 2024-03-24T11:29:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986210+0000", 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986210+0000", 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986212+0000", 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.251e-06 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986219+0000", 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5900000000000002e-06 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986234+0000", 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4783999999999999e-05 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986242+0000", 2024-03-24T11:29:53.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6720000000000004e-06 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986312+0000", 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0051999999999999e-05 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986430+0000", 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011825200000000001 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.986549+0000", 2024-03-24T11:29:53.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119368 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.988936+0000", 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023865319999999998 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:37.988997+0000", 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0937000000000003e-05 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.127634+0000", 2024-03-24T11:29:53.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13863715099999999 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.127699+0000", 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5094999999999999e-05 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.127761+0000", 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1811999999999997e-05 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.462 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:189862 8.8 8:1506632e:::benchmark_data_smithi179_105658_object61820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:38.011249+0000", 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 135.39191636699999, 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.131359372, 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189862 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.463 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011249+0000", 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011249+0000", 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011250+0000", 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.22e-06 2024-03-24T11:29:53.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011255+0000", 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6909999999999996e-06 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011268+0000", 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3360000000000001e-05 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011278+0000", 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4329999999999993e-06 2024-03-24T11:29:53.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011315+0000", 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7567000000000001e-05 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011396+0000", 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1069999999999995e-05 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.011468+0000", 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2193000000000005e-05 2024-03-24T11:29:53.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.013514+0000", 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020451890000000002 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.013711+0000", 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019773699999999999 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.142491+0000", 2024-03-24T11:29:53.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12877997399999999 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.142571+0000", 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9963000000000007e-05 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.142608+0000", 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6974999999999998e-05 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:189876 8.8 8:1ba8f620:::benchmark_data_smithi179_105658_object61834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:38.032902+0000", 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 135.37026291199999, 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109797463, 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189876 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.032902+0000", 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.032902+0000", 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.032903+0000", 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9899999999999999e-07 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.032907+0000", 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3759999999999999e-06 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.032917+0000", 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0053e-05 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.032922+0000", 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.755e-06 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.032970+0000", 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8013999999999998e-05 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.033054+0000", 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3586999999999998e-05 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.033127+0000", 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2562000000000001e-05 2024-03-24T11:29:53.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.035274+0000", 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021475169999999998 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.035480+0000", 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020581899999999999 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.142634+0000", 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107154232 2024-03-24T11:29:53.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.142663+0000", 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9312e-05 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:38.142700+0000", 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6436999999999997e-05 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:71912 9.f 9:fc5e63f1:::benchmark_data_smithi179_130919_object71911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:47.809739+0000", 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5934258459999997, 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104181962, 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 71912 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.809739+0000", 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.809739+0000", 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.809741+0000", 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.229e-06 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.809770+0000", 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9011999999999999e-05 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.809782+0000", 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2262000000000001e-05 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.809791+0000", 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8769999999999999e-06 2024-03-24T11:29:53.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810248+0000", 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00045750599999999999 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810369+0000", 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120399 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810501+0000", 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013195500000000001 2024-03-24T11:29:53.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.812586+0000", 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020850840000000001 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.875986+0000", 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.063400468000000001 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.913849+0000", 2024-03-24T11:29:53.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.03786258 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.913883+0000", 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4493999999999999e-05 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.913921+0000", 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8096000000000003e-05 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:71914 9.6 9:681634b1:::benchmark_data_smithi179_130919_object71913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:47.810947+0000", 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5922187130000003, 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103210145, 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 71914 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.482 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810947+0000", 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810947+0000", 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810947+0000", 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2099999999999995e-07 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810987+0000", 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9668e-05 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.810999+0000", 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1576000000000001e-05 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811006+0000", 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1860000000000003e-06 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811066+0000", 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0167000000000002e-05 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811187+0000", 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120946 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811325+0000", 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013812000000000001 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.813762+0000", 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024366330000000001 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.814132+0000", 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037034900000000001 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914070+0000", 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099938021000000002 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914123+0000", 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3088999999999999e-05 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914157+0000", 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569000000000003e-05 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77589 9.f 9:f55937bc:::benchmark_data_smithi179_130919_object77588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:29:53.487 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.610672+0000", 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.792493106, 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10988160499999999, 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77589 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610672+0000", 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610672+0000", 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610673+0000", 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0699999999999997e-07 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610712+0000", 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8955999999999998e-05 2024-03-24T11:29:53.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610733+0000", 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1308e-05 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610740+0000", 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2219999999999996e-06 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.613294+0000", 2024-03-24T11:29:53.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002554115 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.613372+0000", 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7207999999999997e-05 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.613488+0000", 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116705 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.617227+0000", 2024-03-24T11:29:53.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037385399999999998 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624474+0000", 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0072472630000000003 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720443+0000", 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095969164999999995 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720500+0000", 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6585000000000001e-05 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720554+0000", 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3931000000000003e-05 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77590 9.f 9:fa033d4e:::benchmark_data_smithi179_130919_object77589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.610755+0000", 2024-03-24T11:29:53.493 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.79240992399999999, 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11337232, 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77590 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610755+0000", 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610755+0000", 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610756+0000", 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3000000000000001e-07 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610776+0000", 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9698e-05 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610781+0000", 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4650000000000002e-06 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.610785+0000", 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6940000000000001e-06 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615214+0000", 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044294529999999999 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615329+0000", 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114406 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615450+0000", 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121061 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.619414+0000", 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039645920000000003 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720343+0000", 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100928565 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.724020+0000", 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003677479 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.724073+0000", 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2602999999999998e-05 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.724128+0000", 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4873999999999997e-05 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77593 9.f 9:f761f1e4:::benchmark_data_smithi179_130919_object77592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.615349+0000", 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.78781619199999997, 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10890722999999999, 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77593 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615349+0000", 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615349+0000", 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615350+0000", 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3680000000000001e-06 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615398+0000", 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7246999999999997e-05 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615407+0000", 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0729999999999994e-06 2024-03-24T11:29:53.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.615413+0000", 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7950000000000004e-06 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.617257+0000", 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018445079999999999 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.617422+0000", 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016448999999999999 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.617565+0000", 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000143077 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.619539+0000", 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001974328 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720393+0000", 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10085384999999999 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.724175+0000", 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003781782 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.724222+0000", 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7509000000000002e-05 2024-03-24T11:29:53.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.724256+0000", 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4202999999999998e-05 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77599 9.6 9:67a61705:::benchmark_data_smithi179_130919_object77598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.618195+0000", 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.78497062699999998, 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105086079, 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77599 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618195+0000", 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618195+0000", 2024-03-24T11:29:53.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618196+0000", 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0419999999999999e-06 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618245+0000", 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9599999999999999e-05 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618255+0000", 2024-03-24T11:29:53.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3070000000000008e-06 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618260+0000", 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7219999999999996e-06 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618312+0000", 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1771999999999998e-05 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618393+0000", 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0547e-05 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.618477+0000", 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4695000000000001e-05 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.620399+0000", 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019217080000000001 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624384+0000", 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039853170000000004 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.723178+0000", 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098793417999999994 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.723235+0000", 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7083000000000002e-05 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.723281+0000", 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5868000000000002e-05 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77601 9.6 9:60f49282:::benchmark_data_smithi179_130919_object77600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.624911+0000", 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.77825481200000002, 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113272737, 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.511 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77601 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624911+0000", 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624911+0000", 2024-03-24T11:29:53.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624911+0000", 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5099999999999998e-07 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624977+0000", 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5707e-05 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624984+0000", 2024-03-24T11:29:53.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4050000000000003e-06 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.624991+0000", 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4649999999999999e-06 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.515 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 45 16 78540 78524 109.043 68.875 0.0107088 0.00916503 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625020+0000", 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8634000000000001e-05 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625106+0000", 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6648000000000001e-05 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2", 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720646+0000", 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095540066000000007 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.726169+0000", 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055226820000000001 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.726189+0000", 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9947000000000001e-05 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738093+0000", 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011903904999999999 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738142+0000", 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9499000000000002e-05 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738183+0000", 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0927999999999998e-05 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77602 9.a 9:5d176224:::benchmark_data_smithi179_130919_object77601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.625001+0000", 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.77816387499999995, 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112894073, 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77602 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.518 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625001+0000", 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625001+0000", 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625002+0000", 2024-03-24T11:29:53.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4400000000000001e-07 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625027+0000", 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5202e-05 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625032+0000", 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9150000000000004e-06 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625035+0000", 2024-03-24T11:29:53.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4589999999999998e-06 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625076+0000", 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0065999999999999e-05 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625158+0000", 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2817000000000003e-05 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720479+0000", 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095320594999999994 2024-03-24T11:29:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.728676+0000", 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081966899999999995 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.728686+0000", 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0575e-05 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.728718+0000", 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1417000000000001e-05 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.737845+0000", 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0091269939999999994 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.737896+0000", 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0899000000000002e-05 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.523 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.15357.0:77599 9.6 e228/211 ondisk, result = 0)", 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.625272+0000", 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.77789318399999996, 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098015999000000006, 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625272+0000", 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625272+0000", 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625273+0000", 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3099999999999998e-07 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.625275+0000", 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.418e-06 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:53.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720361+0000", 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095086282999999994 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.720370+0000", 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0950000000000002e-06 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.723165+0000", 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027944189999999998 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.723172+0000", 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.362e-06 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.723288+0000", 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011599099999999999 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:53.527 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:29:53.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:53.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:53.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:53.691 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:53.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:53.754 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:29:53.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:29:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:29:53.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:53.833 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:53.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:53.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:53.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:53.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:54.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:54.108 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:54.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:53 smithi181 ceph-mon[33998]: pgmap v458: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 713 op/s 2024-03-24T11:29:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1800850318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:53 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:53 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:29:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:53 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:29:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[29608]: pgmap v458: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 713 op/s 2024-03-24T11:29:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1800850318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:54.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:29:54.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:29:54.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:54.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[37907]: pgmap v458: 168 pgs: 168 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 713 op/s 2024-03-24T11:29:54.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1800850318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:54.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:29:54.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:29:54.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:53 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:29:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:54.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:54.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:54.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:29:54.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:54.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:54.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:54.501 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 46 16 79820 79804 108.412 80 0.00723725 0.00921993 2024-03-24T11:29:54.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:54.557 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:54.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:54.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:54.710 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:54.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:54.740 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:54.750 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:54.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:54.848 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:54.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:54.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:54.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:54.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:55.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:55.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:55.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:29:55.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[29608]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T11:29:55.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[29608]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T11:29:55.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:55.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:55.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:29:55.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[37907]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T11:29:55.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:54 smithi179 ceph-mon[37907]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T11:29:55.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:55.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:55.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:29:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:54 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:29:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:54 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:29:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:54 smithi181 ceph-mon[33998]: osdmap e229: 8 total, 8 up, 8 in 2024-03-24T11:29:55.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:54 smithi181 ceph-mon[33998]: osdmap e230: 8 total, 8 up, 8 in 2024-03-24T11:29:55.222 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:55.287 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:55.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:55.328 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:29:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:29:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:29:55.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:55.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:55.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:55.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 47 16 81084 81068 107.786 79 0.0156371 0.00927316 2024-03-24T11:29:55.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:55.562 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:55.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:55.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:55.702 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:55.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:55.771 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:55 smithi181 ceph-mon[33998]: pgmap v461: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-03-24T11:29:55.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:29:55.848 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:55.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:29:55.901 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:55 smithi179 ceph-mon[29608]: pgmap v461: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-03-24T11:29:55.902 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:55 smithi179 ceph-mon[37907]: pgmap v461: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-03-24T11:29:55.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:55.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:55.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:56.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:56.132 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:56.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:56.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:56.270 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:56.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:56.371 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:56.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:56.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:56.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 48 16 82418 82402 107.277 83.375 0.00487208 0.00931554 2024-03-24T11:29:56.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:56.557 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:56.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:56.578 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:29:56.578 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:29:56.578 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:29:56.578 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:29:56.578 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:29:56.578 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:29:56.579 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:29:56.579 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:29:56.579 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:29:56.579 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:29:56.579 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:29:56.579 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:29:56.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:56.698 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:56.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:56.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:56.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:56.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:56.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:56.977 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:56.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:57.079 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:57.135 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:57.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:57.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:57.273 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:29:57.278 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:57.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:83502 9.b e230/211 9:da6d07b5:::benchmark_data_smithi179_130919_object83501:head v 230'5239, mlcod=230'5236)", 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:57.336163+0000", 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.003112058, 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031480459999999998, 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:29:57.340 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336163+0000", 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336163+0000", 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336164+0000", 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7800000000000001e-07 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336234+0000", 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9851999999999995e-05 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336248+0000", 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.449e-05 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336255+0000", 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1659999999999997e-06 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336302+0000", 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6792000000000002e-05 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.336320+0000", 2024-03-24T11:29:57.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7856e-05 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:57.339231+0000", 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029105329999999999 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2024-03-24T11:29:57.344 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:29:57.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:57.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:57.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 49 15 83707 83692 106.733 80.625 0.0116029 0.00936478 2024-03-24T11:29:57.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:57.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:57.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:57.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:57.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:57.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:57.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:57.830 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:57.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:29:57.902 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:29:57.902 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-24T11:29:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:57 smithi181 ceph-mon[33998]: pgmap v462: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 722 B/s rd, 53 MiB/s wr, 840 op/s 2024-03-24T11:29:57.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:57.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:57.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:58.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:29:58.123 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:57 smithi179 ceph-mon[29608]: pgmap v462: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 722 B/s rd, 53 MiB/s wr, 840 op/s 2024-03-24T11:29:58.123 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:57 smithi179 ceph-mon[37907]: pgmap v462: 168 pgs: 168 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 722 B/s rd, 53 MiB/s wr, 840 op/s 2024-03-24T11:29:58.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:58.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:29:58.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:29:58.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:29:58.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:58.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:29:58.376 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:29:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:58.429 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:29:58.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:29:58.491 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:58.501 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 50 16 84920 84904 106.114 75.75 0.0134763 0.00941661 2024-03-24T11:29:58.521 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:29:58.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:58.542 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:29:58.571 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:58.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:58.582 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:29:58.621 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:29:58.662 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:29:58.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:58.698 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:29:58.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:58.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:58.742 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:29:58.780 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:29:58.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:58.823 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:29:58.834 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:29:58.845 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:29:58.845 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:29:58.845 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:29:58.845 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:29:58.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:29:58.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:29:58.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:29:58.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:29:58.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:58.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:58.926 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:29:58.926 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:29:58.926 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[29608]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T11:29:58.926 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/147902433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:58.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:29:58.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:29:58.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[37907]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T11:29:58.927 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/147902433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:58.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:58.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:58 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:29:58.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:58 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:29:58.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:58 smithi181 ceph-mon[33998]: osdmap e231: 8 total, 8 up, 8 in 2024-03-24T11:29:58.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/147902433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:29:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:58.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:58.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:59.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:29:59.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:59.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:29:59.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:59.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:59.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:59.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:59.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:59.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:59.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 51 16 86170 86154 105.565 78.125 0.0123428 0.00946792 2024-03-24T11:29:59.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:59.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:59.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:29:59.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:29:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:59.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:59.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:29:59.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:29:59.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:59.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:29:59.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:29:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:29:59.961 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:29:59.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:00.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:00.098 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:59 smithi181 ceph-mon[33998]: pgmap v464: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:30:00.098 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:59 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:30:00.098 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:59 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:30:00.098 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:29:59 smithi181 ceph-mon[33998]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T11:30:00.103 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:00.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:00.205 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:00.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[29608]: pgmap v464: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:30:00.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:30:00.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:30:00.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[29608]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T11:30:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[37907]: pgmap v464: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-24T11:30:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:30:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:30:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:29:59 smithi179 ceph-mon[37907]: osdmap e232: 8 total, 8 up, 8 in 2024-03-24T11:30:00.242 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:00.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:00.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:00.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:00.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:00.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:00.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 52 16 87431 87415 105.05 78.8125 0.0146002 0.00951422 2024-03-24T11:30:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:00.525 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:00.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:00.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:00.634 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:30:00.634 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:30:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:00.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:00.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:00.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:00.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:00.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:00.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:00.952 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:00.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:01.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:01.090 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:01.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:01.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:01.105 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:00 smithi179 ceph-mon[29608]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T11:30:01.105 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:00 smithi179 ceph-mon[29608]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T11:30:01.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:00 smithi179 ceph-mon[37907]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T11:30:01.105 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:00 smithi179 ceph-mon[37907]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T11:30:01.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:00 smithi181 ceph-mon[33998]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T11:30:01.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:00 smithi181 ceph-mon[33998]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T11:30:01.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:01.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:01.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:01.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:01.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:01.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:01.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T11:30:01.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:01.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 53 16 88786 88770 104.666 84.6875 0.0114977 0.00954932 2024-03-24T11:30:01.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:01.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:01.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:01.655 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:01.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:01.756 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:01.792 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:01.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:01.845 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:30:01.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:01.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:01.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:02.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:02.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:02.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:02.148 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:01 smithi181 ceph-mon[33998]: pgmap v466: 168 pgs: 168 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 840 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-24T11:30:02.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:02.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:01 smithi179 ceph-mon[29608]: pgmap v466: 168 pgs: 168 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 840 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-24T11:30:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:01 smithi179 ceph-mon[37907]: pgmap v466: 168 pgs: 168 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 840 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-24T11:30:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:02.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:02.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:02.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:02.365 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:02.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:02.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:02.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 54 16 90083 90067 104.229 81.0625 0.011054 0.00958934 2024-03-24T11:30:02.504 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:02.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:02.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:02.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:02.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:02.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:02.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:02.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:02.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:02.929 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:02.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:03.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:30:03.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:03.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:03.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:03.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:03.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:03.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:03.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:03.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:03.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:03.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:03.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 55 16 91297 91281 103.713 75.875 0.0153679 0.00963747 2024-03-24T11:30:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:03.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:03.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:03.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:03.650 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:03.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:no valid command found; 10 closest matches: 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:0 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:1 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:2 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:abort 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:assert 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:bench [] [] [] [] 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs debug_inject_read_zeros 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs files list 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluefs stats 2024-03-24T11:30:03.714 INFO:teuthology.orchestra.run.smithi179.stderr:bluestore allocator dump block 2024-03-24T11:30:03.715 INFO:teuthology.orchestra.run.smithi179.stderr:Error EINVAL: invalid command 2024-03-24T11:30:03.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:03.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:03.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:03.807 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:30:03.807 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:30:03.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:03 smithi181 ceph-mon[33998]: pgmap v467: 168 pgs: 168 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 802 op/s 2024-03-24T11:30:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:03.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:30:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:30:03.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:03.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:04.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:04.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:04.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:04.094 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:03 smithi179 ceph-mon[29608]: pgmap v467: 168 pgs: 168 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 802 op/s 2024-03-24T11:30:04.094 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:04.096 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:03 smithi179 ceph-mon[37907]: pgmap v467: 168 pgs: 168 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 802 op/s 2024-03-24T11:30:04.096 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:04.173 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:04.206 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:04.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:04.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:04.342 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:04.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:04.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:04.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:04.470 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:04.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:04.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T11:30:04.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 56 16 92438 92422 103.134 71.3125 0.0534572 0.00968908 2024-03-24T11:30:04.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:04.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:04.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:04.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:04.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:04.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:04.832 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:30:04.833 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:04.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:04.904 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:04.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:04.919 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:30:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T11:30:04.978 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-24T11:30:05.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:05.042 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:05.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:05.091 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2697385496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:30:05.091 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:04 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:30:05.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:05.187 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:05.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:05.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:05.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2697385496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:30:05.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:04 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:30:05.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:30:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:30:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2697385496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:30:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:04 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:30:05.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:05.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:05.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:05.427 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:05.467 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:05.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:05.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 57 16 93578 93562 102.575 71.25 0.00566509 0.00974421 2024-03-24T11:30:05.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:05.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:05.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:05.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:05.742 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:05.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:05.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:05.883 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:05.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:05.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:06.021 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:06.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:06.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:05 smithi181 ceph-mon[33998]: pgmap v468: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-24T11:30:06.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:05 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:06.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:05 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:30:06.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:05 smithi181 ceph-mon[33998]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T11:30:06.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:05 smithi181 ceph-mon[33998]: 9.f scrub starts 2024-03-24T11:30:06.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:06.159 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:06.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:06.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[29608]: pgmap v468: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-24T11:30:06.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:06.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:30:06.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[29608]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T11:30:06.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[29608]: 9.f scrub starts 2024-03-24T11:30:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[37907]: pgmap v468: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-03-24T11:30:06.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:06.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:30:06.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[37907]: osdmap e233: 8 total, 8 up, 8 in 2024-03-24T11:30:06.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:05 smithi179 ceph-mon[37907]: 9.f scrub starts 2024-03-24T11:30:06.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:06.314 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:06.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:06.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:06.449 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:06.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:06.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 58 16 94854 94838 102.181 79.75 0.0229933 0.00978233 2024-03-24T11:30:06.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:06.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:06.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:06.707 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:06.741 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:06.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:06.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:06.885 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:06.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:06.893 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:30:06.893 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:30:06.924 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:06.987 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:07.026 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:07.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:07.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:07.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:07.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:07.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:07.302 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:07.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:07.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:07.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:07.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:07.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 59 15 96092 96077 101.762 77.4375 0.00806971 0.00982246 2024-03-24T11:30:07.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:07.575 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:07.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:07.649 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T11:30:07.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:07.716 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:07.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:07.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:07.852 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:07.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:07.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:07 smithi181 ceph-mon[33998]: pgmap v470: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-24T11:30:07.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:07 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:30:07.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:07 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:30:07.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:07 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:30:07.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:07 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:30:07.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:07 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:30:07.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:07 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:30:07.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.008 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:08.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:08.045 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:30:08.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:08.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:08.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[29608]: pgmap v470: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-24T11:30:08.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:30:08.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:30:08.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:30:08.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[37907]: pgmap v470: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:30:08.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:07 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:30:08.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:08.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:08.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:08.380 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.423 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:08.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:08.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:30:08.503206+0000 min lat: 0.00143742 max lat: 0.133698 avg lat: 0.00985704 2024-03-24T11:30:08.503 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:30:08.503 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 60 16 97377 97361 101.403 80.25 0.00659297 0.00985704 2024-03-24T11:30:08.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:08.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:08.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:08.698 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:08.800 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:08.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:30:08.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:08.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:08.954 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:30:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:08.989 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:08.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:08.997 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:30:09.037 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:30:09.077 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:30:09.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:09.122 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:08 smithi179 ceph-mon[29608]: 9.f scrub ok 2024-03-24T11:30:09.122 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:08 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:30:09.122 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:08 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:30:09.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:08 smithi179 ceph-mon[37907]: 9.f scrub ok 2024-03-24T11:30:09.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:08 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:30:09.122 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:08 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:30:09.123 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:30:09.129 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:09.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:09.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:08 smithi181 ceph-mon[33998]: 9.f scrub ok 2024-03-24T11:30:09.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:08 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:30:09.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:08 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:30:09.163 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:30:09.204 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:30:09.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:09.247 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:30:09.252 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:30:09.252 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:30:09.252 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:30:09.252 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:30:09.252 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:30:09.252 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:30:09.252 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:30:09.253 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:30:09.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:09.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:09.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:09.399 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:09.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:09.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:09.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 61 15 98586 98571 100.981 75.625 0.00814974 0.00989864 2024-03-24T11:30:09.522 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:09.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:09.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:09.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:09.675 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:09.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:09.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:09.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:09.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:09.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:09.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:09.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:30:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[29608]: pgmap v471: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:30:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[29608]: 6.1f scrub ok 2024-03-24T11:30:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1454760608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:30:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:30:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:30:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[37907]: pgmap v471: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:30:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[37907]: 6.1f scrub ok 2024-03-24T11:30:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1454760608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:30:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:09 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:30:09.979 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:30:09.979 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:30:09.980 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-24T11:30:09.980 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:30:09.980 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:30:09.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-24T11:30:10.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:10.098 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:10.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:10.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:09 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:30:10.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:09 smithi181 ceph-mon[33998]: pgmap v471: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:30:10.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:09 smithi181 ceph-mon[33998]: 6.1f scrub ok 2024-03-24T11:30:10.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1454760608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:10.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:09 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:30:10.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:09 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:30:10.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:10.221 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:30:10.221 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:30:10.221 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:30:10.222 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:30:10.222 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:30:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:10.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:10.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:10.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:10.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:10.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:10.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:10.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:10.502 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 62 16 99852 99836 100.627 79.0625 0.00732576 0.00993284 2024-03-24T11:30:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:10.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:10.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:10.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:10.666 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:10.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:10.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:10.800 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:10.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:10.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:10.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:10.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:10.977 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:30:10.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:30:10.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:30:10.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:30:10.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:30:10.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:30:10.978 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:30:10.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:10 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:30:10.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T11:30:11.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:11.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:11.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:11.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:11.212 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:30:11.212 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:30:11.212 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:30:11.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:10 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:30:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:11.231 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:11.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:11.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:11.384 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:11.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:11.432 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:30:11.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:11.503 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 63 16 101069 101053 100.237 76.0625 0.0105818 0.00997167 2024-03-24T11:30:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:11.527 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:11.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:11.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:11.662 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:11.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:11.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:11.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:11.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:11.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:11.940 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:11.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:12.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:12.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:12.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: pgmap v472: 168 pgs: 168 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:30:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:30:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:30:12.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:11 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:30:12.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: pgmap v472: 168 pgs: 168 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:30:12.176 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: pgmap v472: 168 pgs: 168 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:30:12.177 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:30:12.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:30:12.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:30:12.178 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:11 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:30:12.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:12.211 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:12.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:12.313 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:12.351 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:12.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:12.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:12.492 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:12.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:12.503 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 64 15 102325 102310 99.8982 78.5625 0.00897952 0.0100058 2024-03-24T11:30:12.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:12.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:12.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:12.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:12.774 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:12.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:12.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:12.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:12.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:12.923 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:30:12.923 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:30:12.923 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:30:12.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:30:12.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:30:12.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:30:12.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:30:12.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:12 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:30:13.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:13.067 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:13.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:13.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:13.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:30:13.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:30:13.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:30:13.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:30:13.180 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:30:13.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:12 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:30:13.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:13.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:13.309 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:13.324 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:30:13.351 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:13.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:13.399 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26050 7.5 7:a02b24c6:::benchmark_data_smithi179_78267_object26049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 535.57693058799998, 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109015764, 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.400 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26050 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819393+0000", 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819394+0000", 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2999999999999999e-07 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819439+0000", 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4925000000000001e-05 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819450+0000", 2024-03-24T11:30:13.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0586e-05 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819455+0000", 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7219999999999996e-06 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819502+0000", 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6295000000000003e-05 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819583+0000", 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0733000000000001e-05 2024-03-24T11:30:13.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.819680+0000", 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7677999999999996e-05 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820508+0000", 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000827445 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.821885+0000", 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001377365 2024-03-24T11:30:13.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928313+0000", 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106428435 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928370+0000", 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6368e-05 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928409+0000", 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9381999999999997e-05 2024-03-24T11:30:13.405 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26055 7.5 7:a6f221c1:::benchmark_data_smithi179_78267_object26054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 535.57563900100001, 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10789344200000001, 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.406 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26055 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820685+0000", 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820686+0000", 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2999999999999999e-07 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820747+0000", 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1148999999999993e-05 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820760+0000", 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820768+0000", 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7610000000000007e-06 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820814+0000", 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6644999999999998e-05 2024-03-24T11:30:13.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.820932+0000", 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117788 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.821060+0000", 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012757900000000001 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.822115+0000", 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010555689999999999 2024-03-24T11:30:13.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.822131+0000", 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5673000000000001e-05 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928469+0000", 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10633761 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928531+0000", 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2549e-05 2024-03-24T11:30:13.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:17.928578+0000", 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7308000000000002e-05 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26494 7.5 7:a4c9ac9c:::benchmark_data_smithi179_78267_object26493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:30:13.412 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:30:13.413 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 535.27305933499997, 2024-03-24T11:30:13.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15363169800000001, 2024-03-24T11:30:13.413 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.413 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.413 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.413 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:30:13.413 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26494 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123265+0000", 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123266+0000", 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1710000000000001e-06 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123350+0000", 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3776000000000006e-05 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123366+0000", 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6779999999999999e-05 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123375+0000", 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.439e-06 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123428+0000", 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2760000000000003e-05 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123535+0000", 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107574 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.123703+0000", 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016783 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.124976+0000", 2024-03-24T11:30:13.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012726930000000001 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125304+0000", 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032807300000000001 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276785+0000", 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.151481123 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276851+0000", 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6204999999999994e-05 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.276896+0000", 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5274000000000001e-05 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:26500 7.5 7:afbe2af8:::benchmark_data_smithi179_78267_object26499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 535.27054772300005, 2024-03-24T11:30:13.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15542166600000001, 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 26500 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125776+0000", 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125777+0000", 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7100000000000011e-07 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125824+0000", 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6388999999999999e-05 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125835+0000", 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1032999999999999e-05 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125840+0000", 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6099999999999997e-06 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125886+0000", 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5711000000000003e-05 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.125998+0000", 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111952 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.126130+0000", 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013231699999999999 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127017+0000", 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00088662500000000004 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.127330+0000", 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000313006 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281101+0000", 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15377119 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281159+0000", 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7759e-05 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:18.281198+0000", 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9103000000000002e-05 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:66218 7.5 7:a1d02a05:::benchmark_data_smithi179_78267_object66217:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:30:13.425 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 508.771931677, 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087797980999999997, 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 66218 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624392+0000", 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624393+0000", 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0869999999999999e-06 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624444+0000", 2024-03-24T11:30:13.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0929e-05 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624454+0000", 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4730000000000006e-06 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624460+0000", 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7180000000000001e-06 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624515+0000", 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5519999999999997e-05 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624671+0000", 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015631299999999999 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624776+0000", 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104378 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.628751+0000", 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039755060000000002 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.686471+0000", 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057719422999999999 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712076+0000", 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025605297999999999 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712140+0000", 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4049999999999998e-05 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712190+0000", 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0285999999999999e-05 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:66219 7.5 7:a6ed2970:::benchmark_data_smithi179_78267_object66218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 508.77183903500003, 2024-03-24T11:30:13.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087858568999999997, 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 66219 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624485+0000", 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624486+0000", 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8199999999999999e-07 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624563+0000", 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6958000000000004e-05 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624571+0000", 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3319999999999992e-06 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.624577+0000", 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1619999999999996e-06 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626484+0000", 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019068620000000001 2024-03-24T11:30:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626623+0000", 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013903099999999999 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2", 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.626786+0000", 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000163096 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.628876+0000", 2024-03-24T11:30:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020895620000000001 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.686625+0000", 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057748762000000002 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712252+0000", 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.025627644000000002 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712307+0000", 2024-03-24T11:30:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4877000000000003e-05 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:21:44.712344+0000", 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6501000000000002e-05 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:36453 9.3 9:c6004c0e:::benchmark_data_smithi179_130919_object36452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:23.721049+0000", 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 49.675275077000002, 2024-03-24T11:30:13.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.083990466999999999, 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 36453 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721049+0000", 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721049+0000", 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721050+0000", 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.31e-06 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721112+0000", 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1419000000000003e-05 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721129+0000", 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6912e-05 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721138+0000", 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4939999999999992e-06 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721199+0000", 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1404999999999999e-05 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721360+0000", 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016049799999999999 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.721507+0000", 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014732600000000001 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.723149+0000", 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016419 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.727142+0000", 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003992382 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.804928+0000", 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.077786021999999996 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.804988+0000", 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0822000000000001e-05 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:23.805039+0000", 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0977000000000003e-05 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:39333 9.3 9:cce4e81a:::benchmark_data_smithi179_130919_object39332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:25.401605+0000", 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 47.994718857000002, 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085309706999999999, 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.443 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 39333 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401605+0000", 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401605+0000", 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401606+0000", 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7800000000000001e-07 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401629+0000", 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2847e-05 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401640+0000", 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1214e-05 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401646+0000", 2024-03-24T11:30:13.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4230000000000002e-06 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401693+0000", 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6598999999999999e-05 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.401827+0000", 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000133984 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.402009+0000", 2024-03-24T11:30:13.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018213699999999999 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.403506+0000", 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001496413 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.404614+0000", 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001107977 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.486803+0000", 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082189640999999994 2024-03-24T11:30:13.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.486865+0000", 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2187999999999995e-05 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:25.486915+0000", 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9505999999999997e-05 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.448 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:70675 9.3 9:c9cc7ea7:::benchmark_data_smithi179_130919_object70674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:46.720513+0000", 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.675810832, 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088364976999999997, 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 70675 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720513+0000", 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720513+0000", 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720515+0000", 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.389e-06 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720576+0000", 2024-03-24T11:30:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1713999999999999e-05 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720590+0000", 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3546999999999999e-05 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720598+0000", 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3100000000000001e-06 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720660+0000", 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1946000000000006e-05 2024-03-24T11:30:13.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720818+0000", 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015805100000000001 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720980+0000", 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016166800000000001 2024-03-24T11:30:13.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:13.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.723790+0000", 2024-03-24T11:30:13.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028106799999999999 2024-03-24T11:30:13.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.724314+0000", 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052334300000000001 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808768+0000", 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084453818 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808831+0000", 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3031e-05 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808878+0000", 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7479999999999999e-05 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:70683 9.3 9:cbf45302:::benchmark_data_smithi179_130919_object70682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:46.724915+0000", 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.671408684999999, 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084123876, 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.455 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 70683 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.724915+0000", 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.724915+0000", 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.724916+0000", 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.06e-07 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.724991+0000", 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4847999999999999e-05 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.725001+0000", 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0015e-05 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.725009+0000", 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9990000000000008e-06 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.725056+0000", 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6804000000000001e-05 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.725174+0000", 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118028 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.725308+0000", 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013463200000000001 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.727755+0000", 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024467999999999998 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808920+0000", 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.081165014999999993 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808959+0000", 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9119999999999998e-05 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.809006+0000", 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7250999999999999e-05 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.809039+0000", 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2758e-05 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.460 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:70686 9.3 9:c53d7897:::benchmark_data_smithi179_130919_object70685:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:46.728385+0000", 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.667938594999999, 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.082670803000000001, 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 70686 2024-03-24T11:30:13.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728385+0000", 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728385+0000", 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728386+0000", 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9899999999999999e-07 2024-03-24T11:30:13.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728454+0000", 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7527000000000006e-05 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728462+0000", 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4530000000000002e-06 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728469+0000", 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0060000000000003e-06 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728517+0000", 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7590999999999999e-05 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728621+0000", 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103897 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.728712+0000", 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0787999999999999e-05 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.730638+0000", 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019269179999999999 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.810192+0000", 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.079553108999999997 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.810988+0000", 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000796424 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.811024+0000", 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.587e-05 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.811056+0000", 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2320999999999998e-05 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.466 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:71915 9.e 9:76fcee08:::benchmark_data_smithi179_130919_object71914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:47.811505+0000", 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.584818839, 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10257931300000001, 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 71915 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811505+0000", 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811505+0000", 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811506+0000", 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.569999999999999e-07 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811528+0000", 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1773999999999999e-05 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811538+0000", 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8619999999999998e-06 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811545+0000", 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6149999999999999e-06 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811604+0000", 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8560999999999997e-05 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811763+0000", 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015900500000000001 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:30:13.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811911+0000", 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000147751 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.814190+0000", 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022792789999999999 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.814304+0000", 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011387099999999999 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914006+0000", 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099702049000000001 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914053+0000", 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6888000000000002e-05 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914084+0000", 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1701e-05 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.472 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77611 9.d 9:b98574d0:::benchmark_data_smithi179_130919_object77610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.627362+0000", 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.768961846, 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101717734, 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77611 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627362+0000", 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627362+0000", 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627367+0000", 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6120000000000001e-06 2024-03-24T11:30:13.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627411+0000", 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4495e-05 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627421+0000", 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3419999999999997e-06 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627426+0000", 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6779999999999997e-06 2024-03-24T11:30:13.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627484+0000", 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7936999999999998e-05 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627619+0000", 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000135122 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627714+0000", 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4195000000000002e-05 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.629698+0000", 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019840050000000001 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.632388+0000", 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026900549999999998 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.728975+0000", 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.096586940999999996 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729031+0000", 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6331e-05 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729080+0000", 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9020999999999997e-05 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77618 9.d 9:bf5bec0b:::benchmark_data_smithi179_130919_object77617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:30:13.478 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.635819+0000", 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.76050541, 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093501861000000006, 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77618 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635819+0000", 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635819+0000", 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635823+0000", 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3080000000000001e-06 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635868+0000", 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5132000000000001e-05 2024-03-24T11:30:13.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635878+0000", 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6180000000000001e-06 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635883+0000", 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5099999999999998e-06 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635921+0000", 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7871999999999999e-05 2024-03-24T11:30:13.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.636013+0000", 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1605000000000001e-05 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.636108+0000", 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5260999999999995e-05 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.637950+0000", 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001841996 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639110+0000", 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00115979 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729216+0000", 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090106674999999997 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729271+0000", 2024-03-24T11:30:13.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4429999999999999e-05 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729320+0000", 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9663999999999997e-05 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77619 9.d 9:b95bc5b0:::benchmark_data_smithi179_130919_object77618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.639948+0000", 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.756375841000001, 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098236130000000005, 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.484 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77619 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639948+0000", 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639948+0000", 2024-03-24T11:30:13.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639949+0000", 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7100000000000001e-07 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640006+0000", 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6892999999999999e-05 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640015+0000", 2024-03-24T11:30:13.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1929999999999999e-06 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640021+0000", 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6130000000000003e-06 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640051+0000", 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0815000000000002e-05 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640136+0000", 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4977999999999999e-05 2024-03-24T11:30:13.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640226+0000", 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9808999999999993e-05 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.641736+0000", 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015096440000000001 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.643416+0000", 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016803429999999999 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738077+0000", 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094660456000000004 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738135+0000", 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8733999999999997e-05 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738184+0000", 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8980999999999997e-05 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.489 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101783 9.3 9:cd124f41:::benchmark_data_smithi179_130919_object101782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.089560+0000", 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.3067635200000001, 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101679564, 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101783 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.490 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089560+0000", 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089560+0000", 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089562+0000", 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.037e-06 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089594+0000", 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2271999999999999e-05 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089604+0000", 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0200000000000001e-05 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089610+0000", 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6590000000000004e-06 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089660+0000", 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9917999999999999e-05 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089793+0000", 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013369400000000001 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089967+0000", 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017335199999999999 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092374+0000", 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024076240000000001 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.095491+0000", 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031172309999999998 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191102+0000", 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095610076000000002 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191189+0000", 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7107999999999993e-05 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191240+0000", 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1393000000000001e-05 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101786 9.3 9:c2e50cf1:::benchmark_data_smithi179_130919_object101785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.092637+0000", 2024-03-24T11:30:13.496 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.3036873070000001, 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098767010000000002, 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101786 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092637+0000", 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092637+0000", 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092638+0000", 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4900000000000004e-07 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092685+0000", 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7375e-05 2024-03-24T11:30:13.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092697+0000", 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2464e-05 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092706+0000", 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1340000000000004e-06 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092771+0000", 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.491e-05 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092913+0000", 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014212599999999999 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.093070+0000", 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015778400000000001 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.095534+0000", 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024635989999999999 2024-03-24T11:30:13.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096623+0000", 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001088583 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191279+0000", 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094656377999999999 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191365+0000", 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5792999999999997e-05 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191404+0000", 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8915000000000003e-05 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101795 9.3 9:cf9e19dc:::benchmark_data_smithi179_130919_object101794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.096719+0000", 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.2996051770000001, 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094862295999999999, 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.502 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101795 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096719+0000", 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096719+0000", 2024-03-24T11:30:13.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096729+0000", 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7540000000000001e-06 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096798+0000", 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9146999999999994e-05 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096813+0000", 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5288999999999999e-05 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096821+0000", 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4530000000000002e-06 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096876+0000", 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5024e-05 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.097004+0000", 2024-03-24T11:30:13.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012798700000000001 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.097143+0000", 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013871200000000001 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.099660+0000", 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025171 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.103339+0000", 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003678527 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191472+0000", 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088132919000000004 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191546+0000", 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3770999999999996e-05 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191581+0000", 2024-03-24T11:30:13.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5612999999999999e-05 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101798 9.3 9:ca9b5007:::benchmark_data_smithi179_130919_object101797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.100748+0000", 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.2955759900000001, 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090983586000000005, 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.508 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101798 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100748+0000", 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100748+0000", 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100749+0000", 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2759999999999999e-06 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100828+0000", 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8549999999999998e-05 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100844+0000", 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5676999999999999e-05 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100851+0000", 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1420000000000004e-06 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100897+0000", 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6227999999999999e-05 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.101005+0000", 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107911 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.101104+0000", 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9347999999999998e-05 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.103427+0000", 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002322668 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.107710+0000", 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042832879999999997 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191637+0000", 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.083927392000000003 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191689+0000", 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1187000000000003e-05 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191732+0000", 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2919e-05 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101806 9.3 9:c1626947:::benchmark_data_smithi179_130919_object101805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.104985+0000", 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.291339257, 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.086930231999999996, 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101806 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:30:13.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.104985+0000", 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.104985+0000", 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.104986+0000", 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4100000000000001e-06 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.105063+0000", 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7201999999999997e-05 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.105075+0000", 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1907e-05 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.105081+0000", 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0340000000000002e-06 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:30:13.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.105134+0000", 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2519999999999999e-05 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.105274+0000", 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014010800000000001 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.105432+0000", 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015849399999999999 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.107733+0000", 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023007320000000002 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.115739+0000", 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0080058850000000008 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191782+0000", 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.076042824999999994 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191875+0000", 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3618999999999999e-05 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191915+0000", 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9496000000000003e-05 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:30:13.519 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:30:13.520 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 65 16 103585 103569 99.5704 78.6875 0.0146999 0.0100389 2024-03-24T11:30:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:13.541 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:13.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:13.572 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:13.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:13.674 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:13.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:13.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:13.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:13.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:13.885 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:30:13.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 6.b scrub starts 2024-03-24T11:30:13.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 6.b scrub ok 2024-03-24T11:30:13.887 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: pgmap v473: 168 pgs: 168 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:13.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:13.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:13.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:14.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:14.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:14.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:30:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:30:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:30:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:30:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:30:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:13 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:30:14.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 6.b scrub starts 2024-03-24T11:30:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 6.b scrub ok 2024-03-24T11:30:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: pgmap v473: 168 pgs: 168 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:30:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:30:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:30:14.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:30:14.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:30:14.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 6.b scrub starts 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 6.b scrub ok 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: pgmap v473: 168 pgs: 168 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:30:14.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:13 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:30:14.218 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:14.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:14.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:14.363 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:14.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:14.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:14.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:14.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:14.504 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 66 15 104765 104750 99.1801 73.8125 0.0191221 0.0100781 2024-03-24T11:30:14.600 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:14.642 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:14.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:14.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:14.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:14.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:14.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:14.916 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:14.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:15.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:15.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:15.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:15.143 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:30:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 4.11 scrub starts 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 4.11 scrub ok 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 4.11 scrub starts 2024-03-24T11:30:15.144 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 4.11 scrub ok 2024-03-24T11:30:15.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:15.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:30:15.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:30:15.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:30:15.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:30:15.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:30:15.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:14 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:30:15.150 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:15.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 4.11 scrub starts 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 4.11 scrub ok 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:30:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:14 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:30:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:15.200 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:15.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:15.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:30:15.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:15.332 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:15.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:15.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:15.462 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:15.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:15.506 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 67 15 106092 106077 98.9343 82.9375 0.0159517 0.0101031 2024-03-24T11:30:15.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:15.619 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:15.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:15.720 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:15.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:15.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:15.784 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:15.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:15.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:15.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:16.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:16.030 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 4.17 scrub starts 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 4.17 scrub ok 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: pgmap v474: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 4.17 scrub starts 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 4.17 scrub ok 2024-03-24T11:30:16.031 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: pgmap v474: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T11:30:16.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:30:16.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:30:16.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:30:16.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:30:16.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:30:16.032 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:15 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:30:16.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:16.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 4.17 scrub starts 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 4.17 scrub ok 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: pgmap v474: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:30:16.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:15 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:30:16.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:16.270 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:16.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:16.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.460 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:16.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:16.506 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 68 16 107249 107233 98.5418 72.25 0.0218922 0.0101432 2024-03-24T11:30:16.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:16.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:16.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:16.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:16.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:16.869 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:16.912 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:30:16.971 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:16.998 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:16.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:17.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:17.138 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:17.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:30:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:30:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:30:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:30:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:30:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:30:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/123238758' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:30:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:30:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:30:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:30:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:30:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:30:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:16 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/123238758' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:17.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:17.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:17.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:17.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:17.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:16 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:30:17.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:16 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:30:17.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:16 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:30:17.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:16 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:30:17.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:16 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:30:17.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:16 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:30:17.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:16 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/123238758' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:17.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:17.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:17.506 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 69 16 108220 108204 97.9932 60.6875 0.00791724 0.0102001 2024-03-24T11:30:17.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:17.554 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:17.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:17.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:17.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:17.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:17.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:17.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:17.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:17.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:17.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:17.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:18.067 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:18.106 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:18.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:18.174 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: pgmap v475: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 895 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/123238758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: pgmap v475: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 895 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T11:30:18.175 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:30:18.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:30:18.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:30:18.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:30:18.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/123238758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:30:18.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T11:30:18.176 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:17 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:30:18.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:18.240 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:18.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:18.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:18.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:18.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:18.402 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:30:18.402 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T11:30:18.402 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:30:18.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: pgmap v475: 168 pgs: 168 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 895 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T11:30:18.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:30:18.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:30:18.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:30:18.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:30:18.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/123238758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:30:18.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: osdmap e234: 8 total, 8 up, 8 in 2024-03-24T11:30:18.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:17 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:30:18.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:18.506 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 70 15 109452 109437 97.6941 77.0625 0.00915768 0.0102313 2024-03-24T11:30:18.514 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:18.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:18.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:18.655 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:18.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:18.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:18.795 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:18.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:18.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:18.927 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:18.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:19.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:19.067 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:19.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:19.158 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:19.168 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: 6.2 deep-scrub starts 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: 6.2 deep-scrub ok 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: pgmap v477: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:30:19.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: 6.2 deep-scrub starts 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: 6.2 deep-scrub ok 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: pgmap v477: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:30:19.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:19 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:30:19.225 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:19.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:19.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:19.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:19.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: 6.2 deep-scrub starts 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: 6.2 deep-scrub ok 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: pgmap v477: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:30:19.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:19 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:30:19.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:19.504 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:19.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:19.506 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 71 16 110805 110789 97.5082 84.5 0.00854865 0.0102506 2024-03-24T11:30:19.606 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:19.664 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:19.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:19.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:19.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:19.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:19.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:19.935 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:19.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:19.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:20.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:20.084 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:20.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:20.185 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:20.226 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:20.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:20.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:20.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:20.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:20 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:30:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:20 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:30:20.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:20 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:30:20.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:20 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:30:20.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:20.506 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 72 16 112150 112134 97.3214 84.0625 0.0147547 0.0102704 2024-03-24T11:30:20.511 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:20.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:20.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:20 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:30:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:20 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:30:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:20.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:20.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:20.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:20.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:20.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:20.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:20.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:20.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:20.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:30:21.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:21.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:21.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:21.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:21.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:21.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:21.308 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:21.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:21.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:21.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[37907]: 3.a deep-scrub starts 2024-03-24T11:30:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[37907]: 3.a deep-scrub ok 2024-03-24T11:30:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:30:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:30:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[37907]: pgmap v478: 168 pgs: 168 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-24T11:30:21.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[29608]: 3.a deep-scrub starts 2024-03-24T11:30:21.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[29608]: 3.a deep-scrub ok 2024-03-24T11:30:21.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:30:21.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:30:21.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:21 smithi179 ceph-mon[29608]: pgmap v478: 168 pgs: 168 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-24T11:30:21.491 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:21.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:21.506 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 73 15 113437 113422 97.0909 80.5 0.012156 0.0102953 2024-03-24T11:30:21.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:21.636 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:21.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:21 smithi181 ceph-mon[33998]: 3.a deep-scrub starts 2024-03-24T11:30:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:21 smithi181 ceph-mon[33998]: 3.a deep-scrub ok 2024-03-24T11:30:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:21 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:30:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:21 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:30:21.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:21 smithi181 ceph-mon[33998]: pgmap v478: 168 pgs: 168 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-24T11:30:21.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:21.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:21.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:21.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:21.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:21.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:22.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:22.060 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:22.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:22.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:22.196 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:30:22.196 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:22.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:22.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:22.332 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:22.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:22.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:30:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:30:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[37907]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T11:30:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:30:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:30:22.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:30:22.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[29608]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T11:30:22.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:30:22.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:22 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:30:22.470 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:22.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:22.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 74 16 114812 114796 96.9392 85.875 0.0154888 0.0103113 2024-03-24T11:30:22.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:22.609 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:22.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:22.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:22 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:30:22.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:22 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:30:22.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:22 smithi181 ceph-mon[33998]: osdmap e235: 8 total, 8 up, 8 in 2024-03-24T11:30:22.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:22 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:30:22.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:22 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:30:22.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:22.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:22.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:22.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:22.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:22.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:22.897 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:30:22.946 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:30:22.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:22.989 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:30:23.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:23.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:23.024 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:30:23.065 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:30:23.106 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:30:23.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:23.139 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:30:23.156 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:23.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:23.185 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:30:23.193 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:30:23.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:23.287 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:23.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:23.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:23.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:23.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:23.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 75 16 115982 115966 96.6216 73.125 0.00809288 0.0103453 2024-03-24T11:30:23.527 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:23.565 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: 4.d deep-scrub starts 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: 4.d deep-scrub ok 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: pgmap v480: 168 pgs: 168 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 745 op/s 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1360529399' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:30:23.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:30:23.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: 4.d deep-scrub starts 2024-03-24T11:30:23.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: 4.d deep-scrub ok 2024-03-24T11:30:23.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:30:23.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:30:23.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: pgmap v480: 168 pgs: 168 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 745 op/s 2024-03-24T11:30:23.567 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1360529399' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:23.572 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:23.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:30:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:30:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: 4.d deep-scrub starts 2024-03-24T11:30:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: 4.d deep-scrub ok 2024-03-24T11:30:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:30:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:30:23.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: pgmap v480: 168 pgs: 168 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 745 op/s 2024-03-24T11:30:23.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1360529399' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:23.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:23.709 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:23.719 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:23.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:23.778 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:30:23.778 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:30:23.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:23.865 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:23.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:23.967 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:24.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:24.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:24.145 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:24.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:30:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:30:24.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:24.278 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:24.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:24.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[37907]: 9.6 scrub starts 2024-03-24T11:30:24.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[37907]: 6.1c scrub starts 2024-03-24T11:30:24.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[37907]: 6.1c scrub ok 2024-03-24T11:30:24.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[37907]: 3.1e scrub starts 2024-03-24T11:30:24.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[37907]: 3.1e scrub ok 2024-03-24T11:30:24.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:24.384 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[29608]: 9.6 scrub starts 2024-03-24T11:30:24.384 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[29608]: 6.1c scrub starts 2024-03-24T11:30:24.384 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[29608]: 6.1c scrub ok 2024-03-24T11:30:24.384 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[29608]: 3.1e scrub starts 2024-03-24T11:30:24.384 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[29608]: 3.1e scrub ok 2024-03-24T11:30:24.384 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:24.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:24.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 76 16 117215 117199 96.3642 77.0625 0.00453316 0.0103726 2024-03-24T11:30:24.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:24.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:24 smithi181 ceph-mon[33998]: 9.6 scrub starts 2024-03-24T11:30:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:24 smithi181 ceph-mon[33998]: 6.1c scrub starts 2024-03-24T11:30:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:24 smithi181 ceph-mon[33998]: 6.1c scrub ok 2024-03-24T11:30:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:24 smithi181 ceph-mon[33998]: 3.1e scrub starts 2024-03-24T11:30:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:24 smithi181 ceph-mon[33998]: 3.1e scrub ok 2024-03-24T11:30:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:24.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:24.693 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:24.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.826 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:24.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:24.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:24.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:24.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:25.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:25.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:25.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:25.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:25.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:30:25.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:25.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:25.303 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:30:25.303 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:30:25.303 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:30:25.303 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[29608]: pgmap v481: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T11:30:25.303 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:30:25.304 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:30:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:30:25.304 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:30:25.304 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:30:25.304 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:30:25.304 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[37907]: pgmap v481: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T11:30:25.304 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:25 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:30:25.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:25.393 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:25.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:25.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:25.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 77 16 118359 118343 96.0412 71.5 0.0031939 0.0104025 2024-03-24T11:30:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:25.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:25.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:25.527 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:30:25.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:30:25.528 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T11:30:25.528 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2024-03-24T11:30:25.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 7 0 2024-03-24T11:30:25.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:25 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:30:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:25 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:30:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:25 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:30:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:25 smithi181 ceph-mon[33998]: pgmap v481: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T11:30:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:25 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:30:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:25.667 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:25.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:25.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:25.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:25.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:25.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:25.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:25.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:26.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:26.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:26.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:26.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:26.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:26.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:26.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:26.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:30:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:30:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:30:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[37907]: 9.6 scrub ok 2024-03-24T11:30:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:30:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:30:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:30:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:30:26.323 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:30:26.323 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:30:26.323 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[29608]: 9.6 scrub ok 2024-03-24T11:30:26.323 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:30:26.323 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:30:26.323 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:30:26.323 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:26 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:30:26.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:26.441 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:26.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:26.484 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:26.513 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 78 15 119535 119520 95.753 73.5625 0.0109894 0.0104391 2024-03-24T11:30:26.545 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:26.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:26.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:26 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:30:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:26 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:30:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:26 smithi181 ceph-mon[33998]: 9.6 scrub ok 2024-03-24T11:30:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:26 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:30:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:26 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:30:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:26 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:30:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:26 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:30:26.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:26.731 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:26.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:26.776 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:26.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:26.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:26.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:26.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:27.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:27.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:27.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:27.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:27.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:27.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:27.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:27.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:27.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 79 16 120701 120685 95.4626 72.8125 0.00960933 0.0104708 2024-03-24T11:30:27.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:27.649 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:27.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:27 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:30:27.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:27 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:30:27.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:27 smithi181 ceph-mon[33998]: 1.b scrub starts 2024-03-24T11:30:27.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:27 smithi181 ceph-mon[33998]: 1.b scrub ok 2024-03-24T11:30:27.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:27 smithi181 ceph-mon[33998]: pgmap v482: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:30:27.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:27 smithi181 ceph-mon[33998]: 3.12 scrub starts 2024-03-24T11:30:27.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:27 smithi181 ceph-mon[33998]: 3.12 scrub ok 2024-03-24T11:30:27.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:30:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:30:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[37907]: 1.b scrub starts 2024-03-24T11:30:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[37907]: 1.b scrub ok 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[37907]: pgmap v482: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[37907]: 3.12 scrub starts 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[37907]: 3.12 scrub ok 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[29608]: 1.b scrub starts 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[29608]: 1.b scrub ok 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[29608]: pgmap v482: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[29608]: 3.12 scrub starts 2024-03-24T11:30:27.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:27 smithi179 ceph-mon[29608]: 3.12 scrub ok 2024-03-24T11:30:27.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:27.819 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:27.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:27.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:27.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:28.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:28.091 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:28.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:28.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:28.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:28.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:28.259 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:30:28.328 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:28.369 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:28.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:28.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:30:28.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:30:28.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: 9.4 scrub starts 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1194842271' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: 5.7 deep-scrub starts 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[29608]: 5.7 deep-scrub ok 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: 9.4 scrub starts 2024-03-24T11:30:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1194842271' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:30:28.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: 5.7 deep-scrub starts 2024-03-24T11:30:28.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:28 smithi179 ceph-mon[37907]: 5.7 deep-scrub ok 2024-03-24T11:30:28.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:28.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:28.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:28.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:28.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:30:28.508031+0000 min lat: 0.00143742 max lat: 0.133698 avg lat: 0.0105006 2024-03-24T11:30:28.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:30:28.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 80 16 121882 121866 95.1919 73.8125 0.00762507 0.0105006 2024-03-24T11:30:28.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:28.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:28.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:28.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:30:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:30:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:30:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:30:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: 9.4 scrub starts 2024-03-24T11:30:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1194842271' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:30:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: 5.7 deep-scrub starts 2024-03-24T11:30:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:28 smithi181 ceph-mon[33998]: 5.7 deep-scrub ok 2024-03-24T11:30:28.720 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:28.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:28.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:28.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:28.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:28.915 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:28.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:29.017 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:29.054 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:29.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:29.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:29.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:29.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:29.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:29.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:29.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:29.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:29.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:29.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:30:29.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:30:29.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:29.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1194842271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[29608]: pgmap v484: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[29608]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1194842271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[37907]: pgmap v484: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s 2024-03-24T11:30:29.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:29 smithi179 ceph-mon[37907]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T11:30:29.464 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:29.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:29.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 81 16 123273 123257 95.0899 86.9375 0.00492654 0.010512 2024-03-24T11:30:29.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:29.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:29.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:29 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:30:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:29 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:30:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:29 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:29 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1194842271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:30:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:29 smithi181 ceph-mon[33998]: pgmap v484: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s 2024-03-24T11:30:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:29 smithi181 ceph-mon[33998]: osdmap e236: 8 total, 8 up, 8 in 2024-03-24T11:30:29.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:29.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:29.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:29.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:29.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:29.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:29.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:30.013 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:30.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:30.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:30.249 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:30.297 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:30.399 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:30.445 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:30.507 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 82 16 124626 124610 94.9614 84.5625 0.0126973 0.0105263 2024-03-24T11:30:30.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:30.588 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:30.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:30.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:30.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:30.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:30.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:30.902 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:30:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=cleanup t=2024-03-24T11:30:30.488300347Z level=info msg="Completed cleanup jobs" duration=7.483112ms 2024-03-24T11:30:30.902 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:30:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore.transactions t=2024-03-24T11:30:30.4971954Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:30:30.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:30.997 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:30.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:31.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:31.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:31.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:31.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:30:31.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:31.280 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:31.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:31.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:31.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:31.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:31.509 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 83 16 125936 125920 94.8021 81.875 0.00567403 0.0105439 2024-03-24T11:30:31.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:31.558 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:31.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:31.580 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:31 smithi179 ceph-mon[29608]: pgmap v485: 168 pgs: 168 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-24T11:30:31.580 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:31 smithi179 ceph-mon[37907]: pgmap v485: 168 pgs: 168 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-24T11:30:31.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:31 smithi181 ceph-mon[33998]: pgmap v485: 168 pgs: 168 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-24T11:30:31.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:31.696 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:31.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:31.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:31.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:31.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:31.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:31.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:31.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:32.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:32.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:32.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:32.221 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:32.265 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:32.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:32.315 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:30:32.315 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:30:32.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:32.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:32.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:32.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:32.510 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 84 16 127194 127178 94.6086 78.625 0.0177523 0.0105657 2024-03-24T11:30:32.556 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:32.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:32.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:32.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:32.800 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:32.838 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:32.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:32.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:32.977 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:32.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:33.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:33.114 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:33.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:33.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:33.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:33.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:33.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:33.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:33.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:33.492 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:33.510 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 85 16 128425 128409 94.4006 76.9375 0.012206 0.0105888 2024-03-24T11:30:33.530 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:33.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:33.579 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:33 smithi181 ceph-mon[33998]: pgmap v486: 168 pgs: 168 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-24T11:30:33.579 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:33.579 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:33.633 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:33.671 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:33.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:33 smithi179 ceph-mon[29608]: pgmap v486: 168 pgs: 168 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-24T11:30:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:33.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:33 smithi179 ceph-mon[37907]: pgmap v486: 168 pgs: 168 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-24T11:30:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:33.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:33.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:33.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:33.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:30:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:30:33.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:33.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:33.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:34.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.079 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:34.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:34.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:34.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:34.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.356 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:34.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:34.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:34.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:34.510 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 86 16 129536 129520 94.1103 69.4375 0.00386683 0.0106086 2024-03-24T11:30:34.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:34.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:34 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1427552103' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-24T11:30:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:34 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-24T11:30:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:34.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:34 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1427552103' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-24T11:30:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:34 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-24T11:30:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:34 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1427552103' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-24T11:30:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:34 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-24T11:30:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:34.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:34.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:34.859 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:34.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:34.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:34.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:35.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:35.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:35.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:35.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:35.238 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:35.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:35.336 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:30:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:30:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:30:35.340 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:35.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:35.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:35.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:35.510 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 87 16 129719 129703 93.1601 11.4375 0.0104618 0.0107293 2024-03-24T11:30:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:35.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:35.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:35.603 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:35 smithi179 ceph-mon[29608]: pgmap v487: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:35.604 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:35 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-03-24T11:30:35.604 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:35 smithi179 ceph-mon[29608]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T11:30:35.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:35 smithi179 ceph-mon[37907]: pgmap v487: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:35.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:35 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-03-24T11:30:35.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:35 smithi179 ceph-mon[37907]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T11:30:35.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:35.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:35 smithi181 ceph-mon[33998]: pgmap v487: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T11:30:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:35 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-03-24T11:30:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:35 smithi181 ceph-mon[33998]: osdmap e237: 8 total, 8 up, 8 in 2024-03-24T11:30:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:35.661 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:35.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:35.719 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:30:35.756 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:30:35.761 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:30:35.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:35.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:35.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:35.802 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:30:35.838 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.7 primary-affinity to 0 (802) 2024-03-24T11:30:35.844 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:30:35.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:30:35.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:35.926 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:30:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:35.958 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:35.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:35.967 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:30:36.007 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:30:36.017 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:30:36.018 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:30:36.018 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:30:36.018 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:30:36.018 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:30:36.018 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:30:36.018 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:30:36.018 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:30:36.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:36.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:36.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:36.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:36.234 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:36.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:36.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:36.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:36.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:36.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:36.510 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 88 16 130936 130920 92.9658 76.0625 0.0100963 0.0107521 2024-03-24T11:30:36.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:36.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:36.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:36.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:36 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:30:36.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:36 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:30:36.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:36 smithi181 ceph-mon[33998]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T11:30:36.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2867076762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:36.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:36.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:36.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:30:36.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:30:36.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[29608]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T11:30:36.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2867076762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:30:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:30:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[37907]: osdmap e238: 8 total, 8 up, 8 in 2024-03-24T11:30:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2867076762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:36.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:36.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:36.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:36.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:36.943 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:36.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:37.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:37.089 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:37.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:37.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:37.230 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:37.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:37.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:37.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:37.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:37.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:37.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:37.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:37.514 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 89 16 132292 132276 92.8734 84.75 0.021579 0.0107627 2024-03-24T11:30:37.557 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:37.581 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:37.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:37 smithi181 ceph-mon[33998]: pgmap v490: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T11:30:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:37.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:37.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:37 smithi179 ceph-mon[29608]: pgmap v490: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T11:30:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:37 smithi179 ceph-mon[37907]: pgmap v490: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T11:30:37.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:37.796 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:37.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:37.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:37.941 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:37.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:38.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:38.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:38.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:38.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:38.217 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:38.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:38.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:38.357 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:38.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:38.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:38.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:38.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:38.510 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total time run: 90.0067 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total writes made: 133584 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Write size: 65536 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object size: 65536 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Bandwidth (MB/sec): 92.7597 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Bandwidth: 31.5298 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max bandwidth (MB/sec): 213.312 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min bandwidth (MB/sec): 11.4375 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average IOPS: 1484 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev IOPS: 504.477 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max IOPS: 3413 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min IOPS: 183 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average Latency(s): 0.0107767 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Latency(s): 0.0130421 2024-03-24T11:30:38.511 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max latency(s): 1.0648 2024-03-24T11:30:38.512 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min latency(s): 0.00143742 2024-03-24T11:30:38.514 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T11:30:38.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:38.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:38.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:38.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:38.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:38.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:38.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:38.907 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:38.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:39.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:39.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:30:39.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:39.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:39.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:39.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:39.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:39.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:39.361 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:39 smithi179 ceph-mon[29608]: pgmap v491: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-03-24T11:30:39.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:39.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:39.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.597 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:39.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:39.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:39 smithi181 ceph-mon[33998]: pgmap v491: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-03-24T11:30:39.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:39 smithi179 ceph-mon[37907]: pgmap v491: 168 pgs: 168 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-03-24T11:30:39.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:39.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:39.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:39.870 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:39.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:39.999 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:39.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:40.100 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:40.140 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:40.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:40.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:40.277 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:40.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:40.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:40.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:40.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:40.513 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:40.553 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:40.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:40.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:40.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:40.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:40.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:40.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:40.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:40.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:40.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:40.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:40.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:40.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:41.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:41.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:41.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:41.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:41.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:41.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:41.361 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:41.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:41.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:41.481 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:41.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:41.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:41.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:41.602 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:41 smithi179 ceph-mon[29608]: pgmap v492: 168 pgs: 168 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 89 MiB/s wr, 1.48k op/s; 55 KiB/s, 0 objects/s recovering 2024-03-24T11:30:41.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:41 smithi179 ceph-mon[37907]: pgmap v492: 168 pgs: 168 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 89 MiB/s wr, 1.48k op/s; 55 KiB/s, 0 objects/s recovering 2024-03-24T11:30:41.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:41.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:41.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:41 smithi181 ceph-mon[33998]: pgmap v492: 168 pgs: 168 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 89 MiB/s wr, 1.48k op/s; 55 KiB/s, 0 objects/s recovering 2024-03-24T11:30:41.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:41.770 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:41.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:41.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:41.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:41.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:42.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:42.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:42.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:42.141 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:42.179 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:42.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:42.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:42.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:42.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:42.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:42.464 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:42.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:42.551 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:30:42.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T11:30:42.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:42.640 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:42.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:42.676 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:30:42.676 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:30:42.676 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:142537 7.6 7:60057c3e:::benchmark_data_smithi179_78267_object23343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:00.920786+0000", 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 461.75211723799998, 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18071146900000001, 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 142537 2024-03-24T11:30:42.677 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920786+0000", 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920786+0000", 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.678 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920788+0000", 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.449e-06 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920792+0000", 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9199999999999997e-06 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920802+0000", 2024-03-24T11:30:42.679 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6800000000000005e-06 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.920812+0000", 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.064e-05 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.980033+0000", 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.059220361999999999 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.980125+0000", 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2127999999999996e-05 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.980256+0000", 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013109800000000001 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.985707+0000", 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054516340000000003 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.681 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:00.998268+0000", 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012560449 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:01.101389+0000", 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.103121319 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:01.101439+0000", 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9969000000000002e-05 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.682 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:01.101498+0000", 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8820999999999998e-05 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:148311 7.6 7:64996475:::benchmark_data_smithi179_78267_object29117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:30:42.683 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:07.446179+0000", 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 455.22672441600002, 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18068843000000001, 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 148311 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446179+0000", 2024-03-24T11:30:42.684 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446179+0000", 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446180+0000", 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.189e-06 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446184+0000", 2024-03-24T11:30:42.685 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0199999999999996e-06 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446201+0000", 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6697e-05 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.446210+0000", 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4829999999999999e-06 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.686 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.456052+0000", 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0098422119999999995 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.456161+0000", 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000109466 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.456270+0000", 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108256 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.505129+0000", 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048859800000000002 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.505364+0000", 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000234148 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.626742+0000", 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.121378905 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.688 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.626799+0000", 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7040000000000003e-05 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:07.626868+0000", 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8213999999999994e-05 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:149855 7.6 7:6b469030:::benchmark_data_smithi179_78267_object30661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:30:42.689 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 453.30328019199999, 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20188611300000001, 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149855 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.690 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:30:42.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.691 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.691 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.691 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:30:42.691 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369624+0000", 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.53e-07 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369626+0000", 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3609999999999999e-06 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.692 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369632+0000", 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7640000000000002e-06 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.369637+0000", 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7400000000000004e-06 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.408308+0000", 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.038670910000000003 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.693 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.408440+0000", 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000132261 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.408566+0000", 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126103 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.412115+0000", 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0035489990000000002 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.694 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.420871+0000", 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0087561029999999995 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571373+0000", 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150501522 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571442+0000", 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9660999999999996e-05 2024-03-24T11:30:42.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571510+0000", 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7335999999999994e-05 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:149867 7.6 7:6202a7d3:::benchmark_data_smithi179_78267_object30673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:09.387451+0000", 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 453.28545301299999, 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18884748100000001, 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.696 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 149867 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387451+0000", 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.697 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387451+0000", 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387452+0000", 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.065e-06 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387456+0000", 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4510000000000002e-06 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.698 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387472+0000", 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6110999999999999e-05 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.387481+0000", 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8140000000000007e-06 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.420916+0000", 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.033434868999999999 2024-03-24T11:30:42.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.421066+0000", 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015033900000000001 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.421223+0000", 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015693200000000001 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571551+0000", 2024-03-24T11:30:42.700 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150328132 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.571591+0000", 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9356000000000001e-05 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.576173+0000", 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045823180000000002 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.576227+0000", 2024-03-24T11:30:42.701 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4076999999999997e-05 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:09.576298+0000", 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1017000000000001e-05 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:150547 7.6 7:6af38a5f:::benchmark_data_smithi179_78267_object31353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:10.237125+0000", 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 452.43577908899999, 2024-03-24T11:30:42.702 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16952882499999999, 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 150547 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237125+0000", 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.703 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237125+0000", 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237126+0000", 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1909999999999999e-06 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237130+0000", 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0899999999999998e-06 2024-03-24T11:30:42.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237154+0000", 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4261999999999999e-05 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.237162+0000", 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9589999999999995e-06 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.369979+0000", 2024-03-24T11:30:42.705 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13281731999999999 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.370117+0000", 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013744399999999999 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.370252+0000", 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135425 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.706 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.380524+0000", 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010271481000000001 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.380570+0000", 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6026000000000003e-05 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.406540+0000", 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.025970156000000001 2024-03-24T11:30:42.707 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.406589+0000", 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8982e-05 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.406653+0000", 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4488999999999999e-05 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.708 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:150732 7.6 7:635577ad:::benchmark_data_smithi179_78267_object31538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:10.477882+0000", 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 452.19502134099997, 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16796960499999999, 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 150732 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.709 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477882+0000", 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477882+0000", 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477893+0000", 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0869999999999999e-05 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.710 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477898+0000", 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.169e-06 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477912+0000", 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3733e-05 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.477921+0000", 2024-03-24T11:30:42.711 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8370000000000003e-06 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.567027+0000", 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.089106481000000001 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.567151+0000", 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012334299999999999 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.712 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.567266+0000", 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011532900000000001 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.575882+0000", 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086157690000000006 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.579826+0000", 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039438290000000003 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.713 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.645746+0000", 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.065919926000000004 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.645786+0000", 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0173999999999999e-05 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:10.645852+0000", 2024-03-24T11:30:42.714 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6144999999999998e-05 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:151596 7.6 7:6b62085a:::benchmark_data_smithi179_78267_object32402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:11.452170+0000", 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 451.22073335099998, 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16698094199999999, 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.715 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 151596 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452170+0000", 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452170+0000", 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.716 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452172+0000", 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.421e-06 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452176+0000", 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7319999999999998e-06 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452190+0000", 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3608000000000001e-05 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.717 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.452199+0000", 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4149999999999999e-06 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.464029+0000", 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011830139 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.718 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.464150+0000", 2024-03-24T11:30:42.719 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012094700000000001 2024-03-24T11:30:42.719 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.719 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.719 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.719 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.464269+0000", 2024-03-24T11:30:42.719 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119095 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.468119+0000", 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003850315 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.497652+0000", 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029532553 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.619031+0000", 2024-03-24T11:30:42.720 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.121379203 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.619092+0000", 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1082999999999998e-05 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:11.619151+0000", 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9431000000000001e-05 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.721 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152478 7.6 7:61d14fe2:::benchmark_data_smithi179_78267_object33284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.413643+0000", 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 450.25926101800002, 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17386502400000001, 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152478 2024-03-24T11:30:42.722 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413643+0000", 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413643+0000", 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413644+0000", 2024-03-24T11:30:42.723 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7859999999999999e-06 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413650+0000", 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5300000000000004e-06 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413663+0000", 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2855e-05 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.724 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.413671+0000", 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2539999999999992e-06 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.526208+0000", 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11253737 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.526316+0000", 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000108 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.725 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.526431+0000", 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114227 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532755+0000", 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063247149999999999 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.543894+0000", 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011138830000000001 2024-03-24T11:30:42.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587234+0000", 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.043339487000000003 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587398+0000", 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016432 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587508+0000", 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010965 2024-03-24T11:30:42.727 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152480 7.6 7:6ab9905c:::benchmark_data_smithi179_78267_object33286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.417152+0000", 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 450.25575167, 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.170640984, 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.728 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152480 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417152+0000", 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417152+0000", 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.729 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417154+0000", 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0140000000000001e-06 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417160+0000", 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.02e-06 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417182+0000", 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1991e-05 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.730 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417192+0000", 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0131e-05 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.529357+0000", 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.112164508 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.529470+0000", 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112997 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.731 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.529588+0000", 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000118303 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532801+0000", 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032128619999999999 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.549994+0000", 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017193109000000002 2024-03-24T11:30:42.732 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587518+0000", 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037523749000000002 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587732+0000", 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021424399999999999 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587793+0000", 2024-03-24T11:30:42.733 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1056000000000006e-05 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152481 7.6 7:68b334a5:::benchmark_data_smithi179_78267_object33287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 450.25569213699998, 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.170816564, 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.734 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152481 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.735 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417212+0000", 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4600000000000005e-07 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417243+0000", 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0583999999999997e-05 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417252+0000", 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5100000000000004e-06 2024-03-24T11:30:42.736 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.417259+0000", 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5409999999999997e-06 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532833+0000", 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.115574467 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.532950+0000", 2024-03-24T11:30:42.737 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000117262 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.533064+0000", 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011379699999999999 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.543941+0000", 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010877006999999999 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.738 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.550033+0000", 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006091539 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587801+0000", 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.037767884000000002 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.587965+0000", 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016419699999999999 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.588028+0000", 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3230000000000003e-05 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:152618 7.6 7:6556be39:::benchmark_data_smithi179_78267_object33424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:12.603804+0000", 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 450.069099317, 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.175178431, 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.740 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 152618 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603804+0000", 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.741 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603804+0000", 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603805+0000", 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8299999999999996e-07 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603809+0000", 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8850000000000001e-06 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.742 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603819+0000", 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7259999999999997e-06 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.603826+0000", 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1879999999999996e-06 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.717660+0000", 2024-03-24T11:30:42.743 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.113833881 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.717774+0000", 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000114626 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.717894+0000", 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000119498 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.744 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.724573+0000", 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0066788280000000004 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.730616+0000", 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006043484 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.778702+0000", 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.048085485999999997 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.745 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.778925+0000", 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022367599999999999 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:12.778983+0000", 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7469999999999997e-05 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.746 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153073 7.6 7:65c05f1d:::benchmark_data_smithi179_78267_object33879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.201863+0000", 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 449.47104105099999, 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.184056108, 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153073 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.747 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201863+0000", 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201863+0000", 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201864+0000", 2024-03-24T11:30:42.748 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.522e-06 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201870+0000", 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8810000000000001e-06 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201891+0000", 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0806999999999999e-05 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.749 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.201899+0000", 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0160000000000008e-06 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.249353+0000", 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.047454601999999999 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.249517+0000", 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016394399999999999 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.249652+0000", 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013499200000000001 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.253703+0000", 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040510349999999997 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.256442+0000", 2024-03-24T11:30:42.751 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0027389210000000001 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.385800+0000", 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.129357953 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.385855+0000", 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4255000000000001e-05 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.752 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.385919+0000", 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4179999999999999e-05 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153169 7.6 7:6261c8e3:::benchmark_data_smithi179_78267_object33975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.362517+0000", 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 449.31038634700002, 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.20489932799999999, 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.753 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153169 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362517+0000", 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362517+0000", 2024-03-24T11:30:42.754 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362519+0000", 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5910000000000001e-06 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362523+0000", 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0010000000000003e-06 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.755 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362536+0000", 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3351e-05 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.362545+0000", 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0270000000000001e-06 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.404648+0000", 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.042102939999999998 2024-03-24T11:30:42.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.404769+0000", 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120915 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.404929+0000", 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000160056 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.410256+0000", 2024-03-24T11:30:42.757 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053271070000000002 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.420180+0000", 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099239410000000004 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.567303+0000", 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14712319099999999 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.758 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.567361+0000", 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7936999999999998e-05 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.567417+0000", 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5271e-05 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153604 7.6 7:6abf53f5:::benchmark_data_smithi179_78267_object34410:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.918161+0000", 2024-03-24T11:30:42.759 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 448.754742276, 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18721666100000001, 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153604 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918161+0000", 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.761 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.761 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918161+0000", 2024-03-24T11:30:42.761 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:42.762 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.762 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.762 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918164+0000", 2024-03-24T11:30:42.762 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4090000000000001e-06 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918170+0000", 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0870000000000003e-06 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918184+0000", 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4406999999999999e-05 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.763 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918198+0000", 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3628999999999999e-05 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939774+0000", 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021575596999999998 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939928+0000", 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015411799999999999 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.764 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.940048+0000", 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120003 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.952963+0000", 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012915714999999999 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105236+0000", 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.152272716 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.765 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105256+0000", 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9993e-05 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105314+0000", 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7762e-05 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105378+0000", 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4225000000000003e-05 2024-03-24T11:30:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153605 7.6 7:6583e64f:::benchmark_data_smithi179_78267_object34411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.918217+0000", 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 448.754686381, 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.187972839, 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153605 2024-03-24T11:30:42.767 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918217+0000", 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918217+0000", 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918218+0000", 2024-03-24T11:30:42.768 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0800000000000005e-07 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918221+0000", 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0010000000000002e-06 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918238+0000", 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7331000000000001e-05 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.918248+0000", 2024-03-24T11:30:42.769 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9969999999999993e-06 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.943141+0000", 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.024892986999999998 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.943227+0000", 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6169999999999997e-05 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.770 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.943299+0000", 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1586000000000002e-05 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953002+0000", 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.009702769 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105389+0000", 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15238744000000001 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.771 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105958+0000", 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056881200000000003 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106010+0000", 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2571000000000002e-05 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106190+0000", 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000179667 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.772 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153610 7.6 7:6b61266a:::benchmark_data_smithi179_78267_object34416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.924956+0000", 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 448.74794727800003, 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.181386716, 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153610 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.773 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924956+0000", 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924956+0000", 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.774 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924958+0000", 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4759999999999999e-06 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924963+0000", 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8749999999999999e-06 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924983+0000", 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0017000000000002e-05 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.775 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.924992+0000", 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7479999999999999e-06 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.946269+0000", 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021276994 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.946408+0000", 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013992099999999999 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.776 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.946534+0000", 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012597899999999999 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953037+0000", 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0065029839999999998 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105396+0000", 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15235836999999999 2024-03-24T11:30:42.777 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106249+0000", 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00085298399999999999 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106296+0000", 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7108999999999999e-05 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.778 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106343+0000", 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7259000000000003e-05 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153614 7.6 7:67b9b693:::benchmark_data_smithi179_78267_object34420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.929484+0000", 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 448.74342001700001, 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17737636900000001, 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.779 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153614 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929484+0000", 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929484+0000", 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.780 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929485+0000", 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.381e-06 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929492+0000", 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7750000000000004e-06 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.781 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929509+0000", 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6708e-05 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.929518+0000", 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2790000000000004e-06 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.949499+0000", 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019981298000000002 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.782 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.949635+0000", 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135947 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.949764+0000", 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012860500000000001 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953076+0000", 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003312551 2024-03-24T11:30:42.783 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105403+0000", 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15232652599999999 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106432+0000", 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001029248 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106661+0000", 2024-03-24T11:30:42.784 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00022862800000000001 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106860+0000", 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000199423 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24698.0:153608 7.e e121/92 7:7b7381e4:::benchmark_data_smithi179_78267_object34414:head v 121'9772, mlcod=121'9767)", 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.934510+0000", 2024-03-24T11:30:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 448.73839405699999, 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17072061699999999, 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934510+0000", 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934510+0000", 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.786 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934511+0000", 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.071e-06 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934518+0000", 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6179999999999997e-06 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934533+0000", 2024-03-24T11:30:42.787 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4301000000000001e-05 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.934541+0000", 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1459999999999996e-06 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939257+0000", 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.004716273 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.788 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.939277+0000", 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9683000000000001e-05 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105193+0000", 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16591620500000001 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105230+0000", 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7320000000000002e-05 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.789 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153615 7.6 7:61f98c16:::benchmark_data_smithi179_78267_object34421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:13.936023+0000", 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 448.73688095599999, 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.1709802, 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153615 2024-03-24T11:30:42.790 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936023+0000", 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936023+0000", 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.791 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936024+0000", 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3689999999999999e-06 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936030+0000", 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.524e-06 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936047+0000", 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6991999999999999e-05 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.936055+0000", 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6130000000000004e-06 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953105+0000", 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017049983000000001 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953221+0000", 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011586600000000001 2024-03-24T11:30:42.793 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:13.953346+0000", 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012509199999999999 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105128+0000", 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15178178000000001 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.105410+0000", 2024-03-24T11:30:42.794 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028236199999999998 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106906+0000", 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0014960349999999999 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.106956+0000", 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0071999999999998e-05 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.795 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.107003+0000", 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6511999999999998e-05 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24698.0:153967 7.6 7:6c5dc8d3:::benchmark_data_smithi179_78267_object34773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:23:14.442129+0000", 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 448.230774263, 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.18404717000000001, 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:30:42.796 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24698", 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 153967 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442129+0000", 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442129+0000", 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.797 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442131+0000", 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3540000000000001e-06 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442133+0000", 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6709999999999999e-06 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442141+0000", 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.644e-06 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.798 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.442147+0000", 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5890000000000002e-06 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.607217+0000", 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16507073999999999 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.607383+0000", 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016527600000000001 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.799 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,7", 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.607505+0000", 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012231000000000001 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.613531+0000", 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060264079999999996 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.625923+0000", 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012391154 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.800 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.626059+0000", 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013637600000000001 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.626114+0000", 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5143999999999999e-05 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:23:14.626177+0000", 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2503999999999997e-05 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.801 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.802 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.802 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:30:42.802 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:30:42.802 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:30:42.849 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:42.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:42.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:42.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:42.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:43.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:43.128 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:43.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:43.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:43.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:43.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:43.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:43.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:43.394 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:43.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:43.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:43.533 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:43.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:43.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:43 smithi181 ceph-mon[33998]: pgmap v493: 168 pgs: 168 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 38 MiB/s wr, 661 op/s; 55 KiB/s, 0 objects/s recovering 2024-03-24T11:30:43.671 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:43.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:43 smithi179 ceph-mon[29608]: pgmap v493: 168 pgs: 168 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 38 MiB/s wr, 661 op/s; 55 KiB/s, 0 objects/s recovering 2024-03-24T11:30:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:43 smithi179 ceph-mon[37907]: pgmap v493: 168 pgs: 168 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 38 MiB/s wr, 661 op/s; 55 KiB/s, 0 objects/s recovering 2024-03-24T11:30:43.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:43.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:43.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:43.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:43.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:43.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:44.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:44.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:44.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:44.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:30:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:30:44.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:44.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:44.210 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:44.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:44.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:44.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:44.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:44.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:44.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:44.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:44.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:44.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:44.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:44.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:44.724 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:44.759 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:44.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:44.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:44.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:44.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:45.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:45.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:45.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:45.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:45.178 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:45.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:45.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:45.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:45.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:45.384 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:30:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:30:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:30:45.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:45.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:45.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:45.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:45.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:45.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:45.638 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:45 smithi179 ceph-mon[29608]: pgmap v494: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 31 MiB/s wr, 1.23k op/s; 45 KiB/s, 0 objects/s recovering 2024-03-24T11:30:45.638 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:45 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:45.638 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:45 smithi179 ceph-mon[37907]: pgmap v494: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 31 MiB/s wr, 1.23k op/s; 45 KiB/s, 0 objects/s recovering 2024-03-24T11:30:45.638 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:45 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:45.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:45 smithi181 ceph-mon[33998]: pgmap v494: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 31 MiB/s wr, 1.23k op/s; 45 KiB/s, 0 objects/s recovering 2024-03-24T11:30:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:45 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:30:45.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:45.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:45.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:45.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:45.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:45.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:45.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:46.027 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:46.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:46.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:46.168 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:46.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:46.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:30:46.269 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:46.314 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:46.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:46.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:46.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:46.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:46.586 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:46.630 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:46.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:46.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:46.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:46.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:46.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:46.914 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:46.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:47.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:47.048 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:47.070 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:47.079 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:30:47.127 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:30:47.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:47.176 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:47.187 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:30:47.240 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:30:47.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:47.289 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:30:47.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:47.338 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:30:47.401 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:30:47.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:47.430 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:30:47.452 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:47.453 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:30:47.460 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:30:47.460 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:30:47.460 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:30:47.461 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:30:47.461 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:30:47.461 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:30:47.461 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:30:47.461 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:30:47.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:47.585 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:47.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[29608]: pgmap v495: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 28 MiB/s wr, 1.12k op/s; 41 KiB/s, 0 objects/s recovering 2024-03-24T11:30:47.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1381440120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:47.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2027316983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:47.643 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:47.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[37907]: pgmap v495: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 28 MiB/s wr, 1.12k op/s; 41 KiB/s, 0 objects/s recovering 2024-03-24T11:30:47.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1381440120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:47.643 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2027316983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:47.644 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:47 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:47 smithi181 ceph-mon[33998]: pgmap v495: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 28 MiB/s wr, 1.12k op/s; 41 KiB/s, 0 objects/s recovering 2024-03-24T11:30:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1381440120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:30:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2027316983' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:47 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:30:47.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:47.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:47.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:47.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:47.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:47.988 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:47.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:48.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:48.129 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:48.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:48.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:48.268 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:48.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:48.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:48.411 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:48.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:48.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:48.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:48.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:48.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: Adjusting osd_memory_target on smithi179 to 3248M 2024-03-24T11:30:48.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:48.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:48.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:48.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:48.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:48.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:48.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:30:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: Adjusting osd_memory_target on smithi179 to 3248M 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: osdmap e239: 8 total, 8 up, 8 in 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:30:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: Adjusting osd_memory_target on smithi179 to 3248M 2024-03-24T11:30:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:30:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:30:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:30:48.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:30:48.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:48.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:48.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:48.876 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:48.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:49.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:49.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:49.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:49.254 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:49.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:49.396 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:49.435 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:49.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:49.576 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:49.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:49.721 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:49.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:49 smithi179 ceph-mon[29608]: pgmap v497: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 29 MiB/s wr, 1.23k op/s; 44 KiB/s, 0 objects/s recovering 2024-03-24T11:30:49.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:49 smithi179 ceph-mon[37907]: pgmap v497: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 29 MiB/s wr, 1.23k op/s; 44 KiB/s, 0 objects/s recovering 2024-03-24T11:30:49.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:49.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:49.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:49 smithi181 ceph-mon[33998]: pgmap v497: 168 pgs: 168 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 29 MiB/s wr, 1.23k op/s; 44 KiB/s, 0 objects/s recovering 2024-03-24T11:30:49.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:49.991 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:49.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:50.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:50.129 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:50.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:50.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:50.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:50.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:50.365 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:50.405 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:50.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:50.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:50.543 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:50.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:50.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:50.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:50.678 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:50.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:50.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:50.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:50.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:50.848 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:50 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:30:50.848 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:50 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:30:50.849 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:50 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:30:50.849 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:50 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:30:50.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:50 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:30:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:50 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:30:50.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:50.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:30:50.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:50.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:50.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:51.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:51.098 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:51.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:51.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:51.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:51.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:51.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:51.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:51.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:51.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:51.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:51.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:51.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:51.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:51.750 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:51.791 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:51 smithi179 ceph-mon[29608]: pgmap v498: 168 pgs: 168 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 473 KiB/s wr, 1.31k op/s 2024-03-24T11:30:51.791 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:51 smithi179 ceph-mon[29608]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T11:30:51.792 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:51.792 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:51 smithi179 ceph-mon[37907]: pgmap v498: 168 pgs: 168 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 473 KiB/s wr, 1.31k op/s 2024-03-24T11:30:51.792 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:51 smithi179 ceph-mon[37907]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T11:30:51.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:51.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:51.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:51 smithi181 ceph-mon[33998]: pgmap v498: 168 pgs: 168 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 473 KiB/s wr, 1.31k op/s 2024-03-24T11:30:51.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:51 smithi181 ceph-mon[33998]: osdmap e240: 8 total, 8 up, 8 in 2024-03-24T11:30:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:51.941 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:51.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:52.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:52.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:52.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:52.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:52.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:52.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:52.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:30:52.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:52.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:52.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:52.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:52.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:52.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:52.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:52.638 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:52.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:52.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:52.779 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:52.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:52.882 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:52.940 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:52.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:53.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:53.077 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:53.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:53.178 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:53.209 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:53.221 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:30:53.221 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:30:53.237 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:53.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:53.340 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:53.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:53.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:53.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:53.520 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T11:30:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:53.522 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:53.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:53.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:53.661 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:53.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:53.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:53.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:53.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:53.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:53.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:53 smithi181 ceph-mon[33998]: pgmap v500: 168 pgs: 168 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 770 op/s 2024-03-24T11:30:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1516052224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:30:53.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:30:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:30:53.936 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:53.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:53.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:53 smithi179 ceph-mon[29608]: pgmap v500: 168 pgs: 168 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 770 op/s 2024-03-24T11:30:53.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1516052224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:30:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:53 smithi179 ceph-mon[37907]: pgmap v500: 168 pgs: 168 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 770 op/s 2024-03-24T11:30:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1516052224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:30:54.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:54.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:30:54.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:54.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:54.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:54.230 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:54.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:54.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:54.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:54.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:54.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:54.495 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:54.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:54.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:54.639 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:54.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:54.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1516052224' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:30:54.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:54 smithi179 ceph-mon[37907]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T11:30:54.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:54.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1516052224' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:30:54.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:54 smithi179 ceph-mon[29608]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T11:30:54.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:54.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:54.781 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:54.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:54.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1516052224' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:30:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:54 smithi181 ceph-mon[33998]: osdmap e241: 8 total, 8 up, 8 in 2024-03-24T11:30:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:30:54.919 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:54.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:55.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:55.067 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:55.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:55.208 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:55.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:55.344 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:55.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:55.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:30:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:30:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:30:55.481 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:55.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:55.605 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:55.707 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:55.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.731 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:55.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:55.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:55.871 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:55.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:55 smithi181 ceph-mon[33998]: pgmap v502: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-03-24T11:30:55.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:55 smithi179 ceph-mon[29608]: pgmap v502: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-03-24T11:30:55.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:55 smithi179 ceph-mon[37907]: pgmap v502: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.91k op/s 2024-03-24T11:30:55.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:55.997 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:55.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:56.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:56.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:56.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:56.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:56.269 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:56.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:56.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:56.385 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:56.401 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:56.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:56.404 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T11:30:56.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:56.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:56.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:56.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:56.671 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:56.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:56.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:56.809 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:56.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:56.911 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:56.945 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:56.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:57.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:57.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:57.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:57.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:57.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:57.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:57.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:57.366 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:57.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:57.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:57.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:57.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:57.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:57.639 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:57.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:57.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:57.767 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:57.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:57.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:57 smithi181 ceph-mon[33998]: pgmap v503: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:30:57.902 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:57.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:57.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:57.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:57 smithi179 ceph-mon[37907]: pgmap v503: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:30:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:57 smithi179 ceph-mon[29608]: pgmap v503: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:30:58.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.031 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:58.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:30:58.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:58.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:58.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:58.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:58.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:58.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.441 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:58.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:58.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:58.575 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:58.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:58.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:58.817 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.850 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:58.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:58.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:30:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:58.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:58.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:30:59.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:30:59.126 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:59.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:30:59.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:30:59.269 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:59.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:30:59.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:30:59.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:59.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:30:59.501 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:30:59.548 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:59.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:59.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:30:59.644 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:30:59.649 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:30:59.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:59.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:30:59.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:30:59.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[29608]: pgmap v504: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 914 op/s 2024-03-24T11:30:59.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:30:59.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:30:59.806 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[29608]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T11:30:59.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[37907]: pgmap v504: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 914 op/s 2024-03-24T11:30:59.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:30:59.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:30:59.807 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:30:59 smithi179 ceph-mon[37907]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T11:30:59.823 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:30:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:59.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:30:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:59 smithi181 ceph-mon[33998]: pgmap v504: 168 pgs: 168 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 914 op/s 2024-03-24T11:30:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:59 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:30:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:59 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:30:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:30:59 smithi181 ceph-mon[33998]: osdmap e242: 8 total, 8 up, 8 in 2024-03-24T11:30:59.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:30:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:30:59.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:30:59.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:00.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:00.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:00.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:00.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:00.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:00.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:00.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:00.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:00.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:00.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:00.511 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:00.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:00.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:00.646 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:00 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:31:00.646 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:00 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:31:00.647 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:00 smithi179 ceph-mon[37907]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T11:31:00.647 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:00 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:31:00.647 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:00 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:31:00.647 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:00 smithi179 ceph-mon[29608]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T11:31:00.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:00.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:00.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:00.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:00.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:00.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:00 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:31:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:00 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:31:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:00 smithi181 ceph-mon[33998]: osdmap e243: 8 total, 8 up, 8 in 2024-03-24T11:31:00.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:00.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:01.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:01.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:01.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:01.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:01.186 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:01.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:01.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:01.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:01.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:01.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:01.450 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:01.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:01.552 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:01.592 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:01.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:01.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:01.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:01.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:01 smithi181 ceph-mon[33998]: pgmap v507: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-24T11:31:01.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:01.859 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:01.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:01.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:01 smithi179 ceph-mon[37907]: pgmap v507: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-24T11:31:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:01 smithi179 ceph-mon[29608]: pgmap v507: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-24T11:31:01.960 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:02.001 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:02.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.138 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:02.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:02.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:02.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:02.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:02.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:02.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:02.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:02.653 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:02.686 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:02.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:02.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:02.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:02.955 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:02.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:03.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:03.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:03.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:03.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:03.218 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:03.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:03.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:03.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:03.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:03.455 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:03.498 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:03.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:03.599 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:03.638 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:03.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:03.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:03.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:03.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:03.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:03.887 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:03 smithi179 ceph-mon[29608]: pgmap v508: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 786 op/s 2024-03-24T11:31:03.887 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:03.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:03 smithi179 ceph-mon[37907]: pgmap v508: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 786 op/s 2024-03-24T11:31:03.887 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:03.900 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:03 smithi181 ceph-mon[33998]: pgmap v508: 168 pgs: 168 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 786 op/s 2024-03-24T11:31:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:03.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:31:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:31:03.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:03.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:04.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:04.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:04.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:04.136 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:04.182 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:04.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:04.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:04.317 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:04.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:04.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:04.462 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:04.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:04.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:04.593 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:31:04.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:04.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:04.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:31:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:31:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3590620071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:31:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:31:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:31:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3590620071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:31:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:31:04.726 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:31:04.743 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:04.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:04.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:04.882 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:04.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:04 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:31:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:04 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:31:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3590620071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:31:04.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:05.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:05.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:05.161 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:05.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.302 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:05.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:05.303 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:31:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:31:05.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.441 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:05.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:05.541 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:05.578 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:05.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.719 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:05.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:05.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:05.856 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: pgmap v509: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3590620071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: 1.d scrub starts 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: 1.d scrub ok 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:31:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:05 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:31:05.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: pgmap v509: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3590620071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: 1.d scrub starts 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: 1.d scrub ok 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: pgmap v509: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:31:05.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3590620071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:31:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: osdmap e244: 8 total, 8 up, 8 in 2024-03-24T11:31:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: 1.d scrub starts 2024-03-24T11:31:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: 1.d scrub ok 2024-03-24T11:31:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:31:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:05 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:31:05.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:05.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:06.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:06.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:06.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:06.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:06.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:06.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:06.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:06.403 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:06.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:06.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:06.540 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:06.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:06.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:06.674 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:06.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:06.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:06.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:06.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:06.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:31:06.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:31:06.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:31:06.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:31:06.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[29608]: 9.d scrub starts 2024-03-24T11:31:06.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:31:06.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:31:06.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:31:06.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:31:06.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:31:06.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:31:06.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[37907]: 9.d scrub starts 2024-03-24T11:31:06.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:31:06.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:06 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:31:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:06 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:31:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:06 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:31:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:06 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:31:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:06 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:31:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:06 smithi181 ceph-mon[33998]: 9.d scrub starts 2024-03-24T11:31:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:06 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:31:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:06 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:31:06.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:06.948 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:06.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:07.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:07.100 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:07.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:07.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:07.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:07.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:07.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:07.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:07.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:07.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:07.539 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:07.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:07.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:07.683 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:07.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[29608]: pgmap v511: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-24T11:31:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[29608]: 1.14 scrub starts 2024-03-24T11:31:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[29608]: 1.14 scrub ok 2024-03-24T11:31:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3409871612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[37907]: pgmap v511: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-24T11:31:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[37907]: 1.14 scrub starts 2024-03-24T11:31:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[37907]: 1.14 scrub ok 2024-03-24T11:31:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:07 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3409871612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:07.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:07.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:07.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:31:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:31:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:31:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:31:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:31:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:31:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:31:07.882 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:31:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:07 smithi181 ceph-mon[33998]: pgmap v511: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-24T11:31:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:07 smithi181 ceph-mon[33998]: 1.14 scrub starts 2024-03-24T11:31:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:07 smithi181 ceph-mon[33998]: 1.14 scrub ok 2024-03-24T11:31:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:07 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3409871612' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:07.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:07.958 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:07.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:08.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:08.100 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:08.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:08.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:08.239 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:08.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:08.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:08.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:08.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:08.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:08.515 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:08.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:08.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:08.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:08.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:08.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:08.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:08.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: 1.11 scrub starts 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: 1.11 scrub ok 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: 9.d scrub ok 2024-03-24T11:31:08.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/907983876' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: 1.11 scrub starts 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: 1.11 scrub ok 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: 9.d scrub ok 2024-03-24T11:31:08.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/907983876' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:31:08.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:08 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:31:08.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: 1.11 scrub starts 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: 1.11 scrub ok 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: osdmap e245: 8 total, 8 up, 8 in 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: 9.d scrub ok 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/907983876' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:31:08.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:08 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:31:08.916 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:08.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:09.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:09.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:09.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:09.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:09.184 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:09.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:09.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:09.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:09.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:09.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:09.456 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:09.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:09.501 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:31:09.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:31:09.558 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:09.648 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:09.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:09.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: pgmap v513: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: 5.0 scrub starts 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: 5.0 scrub ok 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:31:09.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: pgmap v513: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s 2024-03-24T11:31:09.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: 5.0 scrub starts 2024-03-24T11:31:09.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: 5.0 scrub ok 2024-03-24T11:31:09.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:31:09.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T11:31:09.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:31:09.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:09 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:31:09.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:09.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:09.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:09.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: pgmap v513: 168 pgs: 168 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: 5.0 scrub starts 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: 5.0 scrub ok 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: osdmap e246: 8 total, 8 up, 8 in 2024-03-24T11:31:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:31:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:09 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:31:09.917 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:09.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:10.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.046 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:10.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:10.147 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.183 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:10.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:10.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:10.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:10.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.447 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:10.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:10.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:10.563 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T11:31:10.583 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:10.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.709 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:10.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:10.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:10.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:10.874 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:31:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:10 smithi181 ceph-mon[33998]: 1.2 scrub starts 2024-03-24T11:31:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:10 smithi181 ceph-mon[33998]: 1.2 scrub ok 2024-03-24T11:31:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:10 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:31:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:10 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:31:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:10 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:31:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:10 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:31:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:10 smithi181 ceph-mon[33998]: 9.a scrub starts 2024-03-24T11:31:10.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[29608]: 1.2 scrub starts 2024-03-24T11:31:10.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[29608]: 1.2 scrub ok 2024-03-24T11:31:10.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:31:10.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[29608]: 9.a scrub starts 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[37907]: 1.2 scrub starts 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[37907]: 1.2 scrub ok 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:31:10.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:10 smithi179 ceph-mon[37907]: 9.a scrub starts 2024-03-24T11:31:10.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:10.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:10.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:11.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:11.121 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:11.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:11.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:11.259 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:11.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:11.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:11.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:11.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:11.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:11.526 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:11.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T11:31:11.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:11.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:11.639 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:11.664 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:11.710 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:11.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:11.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:11.787 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[29608]: pgmap v515: 168 pgs: 168 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-03-24T11:31:11.787 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:31:11.787 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[29608]: 9.7 scrub starts 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[37907]: pgmap v515: 168 pgs: 168 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:31:11.788 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:11 smithi179 ceph-mon[37907]: 9.7 scrub starts 2024-03-24T11:31:11.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:11.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:11.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:11 smithi181 ceph-mon[33998]: pgmap v515: 168 pgs: 168 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-03-24T11:31:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:11 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:31:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:11 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:31:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:11 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:31:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:11 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:31:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:11 smithi181 ceph-mon[33998]: 9.7 scrub starts 2024-03-24T11:31:11.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:11.981 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:11.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:12.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:12.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:12.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:12.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:12.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:12.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:12.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:12.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:12.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:12.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:12.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:12.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:12.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:12.678 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:12.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:12.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:12.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:12.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:12.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:12 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:31:12.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:12 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:31:12.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:12 smithi181 ceph-mon[33998]: 4.1d scrub starts 2024-03-24T11:31:12.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:12 smithi181 ceph-mon[33998]: 4.1d scrub ok 2024-03-24T11:31:12.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:12 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:31:12.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:12 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:31:12.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:12 smithi181 ceph-mon[33998]: 9.a scrub ok 2024-03-24T11:31:12.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:12.957 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:12.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:31:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:31:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[29608]: 4.1d scrub starts 2024-03-24T11:31:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[29608]: 4.1d scrub ok 2024-03-24T11:31:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:31:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:31:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[29608]: 9.a scrub ok 2024-03-24T11:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[37907]: 4.1d scrub starts 2024-03-24T11:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[37907]: 4.1d scrub ok 2024-03-24T11:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:12 smithi179 ceph-mon[37907]: 9.a scrub ok 2024-03-24T11:31:13.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:13.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:13.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:13.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:13.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:13.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:13.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:13.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:13.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:13.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:13.499 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:13.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:13.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:13.633 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:13.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:13.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:13.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:13.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:13.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:13.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[29608]: pgmap v516: 168 pgs: 168 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 846 op/s 2024-03-24T11:31:13.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:31:13.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:31:13.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[29608]: 1.f scrub starts 2024-03-24T11:31:13.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[29608]: 1.f scrub ok 2024-03-24T11:31:13.900 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[29608]: 9.7 scrub ok 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[37907]: pgmap v516: 168 pgs: 168 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 846 op/s 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[37907]: 1.f scrub starts 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[37907]: 1.f scrub ok 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:13 smithi179 ceph-mon[37907]: 9.7 scrub ok 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:13 smithi181 ceph-mon[33998]: pgmap v516: 168 pgs: 168 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 846 op/s 2024-03-24T11:31:13.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:13 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:31:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:13 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:31:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:13 smithi181 ceph-mon[33998]: 1.f scrub starts 2024-03-24T11:31:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:13 smithi181 ceph-mon[33998]: 1.f scrub ok 2024-03-24T11:31:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:13 smithi181 ceph-mon[33998]: 9.7 scrub ok 2024-03-24T11:31:13.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:31:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:31:13.910 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:13.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:14.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:14.048 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:14.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:14.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:14.186 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:14.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:14.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:14.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:14.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:14.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:14.471 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:14.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:14.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:14.615 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:14.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:14.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:14.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:14.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:14.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:14 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:31:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:14 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:31:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:14 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:31:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:14 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:31:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:14 smithi181 ceph-mon[33998]: 9.4 scrub starts 2024-03-24T11:31:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:14.906 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:14.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[29608]: 9.4 scrub starts 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:31:14.943 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:31:14.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:14.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:31:14.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:31:14.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:14 smithi179 ceph-mon[37907]: 9.4 scrub starts 2024-03-24T11:31:15.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:15.042 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:15.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:15.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:15.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:31:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:31:15.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:15.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:15.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:15.438 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:15.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:15.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:15.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:15.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:15.722 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:15.781 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 6.1b scrub starts 2024-03-24T11:31:15.781 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: pgmap v517: 168 pgs: 168 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.59k op/s 2024-03-24T11:31:15.781 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 6.1b scrub ok 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: 9.0 scrub starts 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3444023584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:15.782 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 6.1b scrub starts 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: pgmap v517: 168 pgs: 168 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.59k op/s 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 6.1b scrub ok 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: 9.0 scrub starts 2024-03-24T11:31:15.783 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:15 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3444023584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:15.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:15.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:15.901 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:31:15.901 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 6.1b scrub starts 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: pgmap v517: 168 pgs: 168 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.59k op/s 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 6.1b scrub ok 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: 9.0 scrub starts 2024-03-24T11:31:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:15 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3444023584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:15.955 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:15.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:15.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:16.086 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:16.095 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:31:16.095 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:31:16.096 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:31:16.097 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:31:16.097 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:31:16.097 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:31:16.097 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:31:16.097 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:31:16.097 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:31:16.127 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:16.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:16.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:16.258 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:16.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:16.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:16.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:16.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:16.498 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:16.536 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:16.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:16.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:16.667 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:16.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:16.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:16.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:16.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:16.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:16.904 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:16.942 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:16.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:31:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:31:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[29608]: 6.e scrub starts 2024-03-24T11:31:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[29608]: 6.e scrub ok 2024-03-24T11:31:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[29608]: 1.13 scrub starts 2024-03-24T11:31:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[29608]: 1.13 scrub ok 2024-03-24T11:31:16.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[29608]: 9.4 scrub ok 2024-03-24T11:31:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:31:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:31:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[37907]: 6.e scrub starts 2024-03-24T11:31:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[37907]: 6.e scrub ok 2024-03-24T11:31:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[37907]: 1.13 scrub starts 2024-03-24T11:31:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[37907]: 1.13 scrub ok 2024-03-24T11:31:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:16 smithi179 ceph-mon[37907]: 9.4 scrub ok 2024-03-24T11:31:17.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:17.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:17.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:16 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:31:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:16 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:31:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:16 smithi181 ceph-mon[33998]: 6.e scrub starts 2024-03-24T11:31:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:16 smithi181 ceph-mon[33998]: 6.e scrub ok 2024-03-24T11:31:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:16 smithi181 ceph-mon[33998]: 1.13 scrub starts 2024-03-24T11:31:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:16 smithi181 ceph-mon[33998]: 1.13 scrub ok 2024-03-24T11:31:17.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:16 smithi181 ceph-mon[33998]: 9.4 scrub ok 2024-03-24T11:31:17.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:17.217 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:17.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:17.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:17.359 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:17.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:17.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:17.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:17.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:17.595 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:17.635 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:17.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:17.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:17.774 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:17.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:17.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:17.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:17.911 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:17.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[29608]: pgmap v518: 168 pgs: 168 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-03-24T11:31:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:31:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:31:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[29608]: 9.0 scrub ok 2024-03-24T11:31:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[37907]: pgmap v518: 168 pgs: 168 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-03-24T11:31:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:31:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:31:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:17 smithi179 ceph-mon[37907]: 9.0 scrub ok 2024-03-24T11:31:18.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:18.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:18.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:17 smithi181 ceph-mon[33998]: pgmap v518: 168 pgs: 168 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.47k op/s 2024-03-24T11:31:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:17 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:31:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:17 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:31:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:17 smithi181 ceph-mon[33998]: 9.0 scrub ok 2024-03-24T11:31:18.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:18.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:18.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:18.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:18.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:18.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:18.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:18.462 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:18.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:18.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:18.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:18.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:18.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:18.743 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:18.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:18.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:18.883 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:18.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:31:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:31:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:31:18.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:31:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:31:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:31:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:31:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:18 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:31:18.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:19.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:19.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:19.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:18 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:31:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:18 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:31:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:18 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:31:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:18 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:31:19.156 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:19.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:19.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:19.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:19.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:19.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:19.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:19.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:19.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:31:19.496 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:19.510 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:19.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:19.604 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:19.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:19.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 1.4 scrub starts 2024-03-24T11:31:19.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 1.4 scrub ok 2024-03-24T11:31:19.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: pgmap v519: 168 pgs: 168 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s 2024-03-24T11:31:19.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:31:19.689 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 1.4 scrub starts 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 1.4 scrub ok 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: pgmap v519: 168 pgs: 168 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s 2024-03-24T11:31:19.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:31:19.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:31:19.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:19.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:19.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:19.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:19.870 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:19.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:31:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:19 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:31:19.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:20.023 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:20.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:20.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 1.4 scrub starts 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 1.4 scrub ok 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: pgmap v519: 168 pgs: 168 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.28k op/s 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: osdmap e247: 8 total, 8 up, 8 in 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:31:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:31:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:31:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:31:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:31:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:19 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:31:20.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:31:20.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:20.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:20.257 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:20.316 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:20.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:20.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:20.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:20.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:20.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:20.584 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:20.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:20.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:20.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:20.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:20.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:20.859 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:20.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:20 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:31:20.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:20 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:31:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:20 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:31:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:20 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:31:20.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.009 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:21.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:21.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:21.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:20 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:31:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:20 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:31:21.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:21.279 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:21.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:21.424 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:21.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:21.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:21.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:21.698 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:21.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:21.836 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:21.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:21 smithi179 ceph-mon[29608]: pgmap v521: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T11:31:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:21 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:31:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:21 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:31:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:21 smithi179 ceph-mon[37907]: pgmap v521: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T11:31:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:21 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:31:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:21 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:31:21.969 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:21.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:22.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:22.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:22.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:22.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:21 smithi181 ceph-mon[33998]: pgmap v521: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T11:31:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:21 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:31:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:21 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:31:22.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:22.232 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:22.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:22.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:22.371 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:22.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:22.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:22.518 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:22.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:22.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:22.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:22.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:22.694 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:22.704 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:22.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:22.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:22.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:22.877 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:31:22.877 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:31:22.877 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:31:22.877 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:31:22.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:31:22.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:31:22.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:31:22.878 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:22 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:31:22.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:22.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:22.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:23.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:23.065 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:23.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:22 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:31:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:22 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:31:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:22 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:31:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:22 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:31:23.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:23.209 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:23.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:23.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:23.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:23.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:23.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:23.480 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:23.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:23.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:23.616 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:23.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:23.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:23.737 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:31:23.751 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T11:31:23.768 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:23.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:23.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:23.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:23.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 3.8 scrub starts 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 3.8 scrub ok 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: pgmap v522: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3050284751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/87683443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:31:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 3.8 scrub starts 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 3.8 scrub ok 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: pgmap v522: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3050284751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:31:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/87683443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:31:23.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:23 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:31:24.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:24.040 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:24.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:24.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:24.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 3.8 scrub starts 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 3.8 scrub ok 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: pgmap v522: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3050284751' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:31:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:31:24.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:31:24.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/87683443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:31:24.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:23 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:31:24.180 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:24.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:24.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:24.318 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:24.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:24.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:24.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:24.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:24.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:24.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:24.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:24.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:24.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:24.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:24.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:24.865 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:24.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:24.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:25.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:25.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/87683443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:31:25.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:31:25.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:31:25.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/87683443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:31:25.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T11:31:25.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:25.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:24 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:31:25.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.142 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:25.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/87683443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: osdmap e248: 8 total, 8 up, 8 in 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:24 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:31:25.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:25.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:25.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:25.407 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:31:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:31:25.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:25.545 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:25.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:25.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:25.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:25.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:25.820 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:25.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:25.957 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:25.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[29608]: pgmap v524: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:31:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[29608]: 1.12 deep-scrub starts 2024-03-24T11:31:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[29608]: 1.12 deep-scrub ok 2024-03-24T11:31:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:31:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:31:25.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[37907]: pgmap v524: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:31:25.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[37907]: 1.12 deep-scrub starts 2024-03-24T11:31:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[37907]: 1.12 deep-scrub ok 2024-03-24T11:31:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:31:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:25 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:31:26.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:26.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:26.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:25 smithi181 ceph-mon[33998]: pgmap v524: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:31:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:25 smithi181 ceph-mon[33998]: 1.12 deep-scrub starts 2024-03-24T11:31:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:25 smithi181 ceph-mon[33998]: 1.12 deep-scrub ok 2024-03-24T11:31:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:25 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:31:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:25 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:31:26.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:26.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:26.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:26.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:26.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:26.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:26.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:26.499 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:26.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:26.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:26.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:26.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:26.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:26.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:26.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:26.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:26.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:26.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:27.000 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:27.037 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:27.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:27.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:27.167 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:26 smithi181 ceph-mon[33998]: 1.10 scrub starts 2024-03-24T11:31:27.168 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:26 smithi181 ceph-mon[33998]: 1.10 scrub ok 2024-03-24T11:31:27.168 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:26 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:31:27.168 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:26 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:31:27.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:27.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[29608]: 1.10 scrub starts 2024-03-24T11:31:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[29608]: 1.10 scrub ok 2024-03-24T11:31:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:31:27.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:31:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[37907]: 1.10 scrub starts 2024-03-24T11:31:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[37907]: 1.10 scrub ok 2024-03-24T11:31:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:31:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:26 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:31:27.306 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:27.343 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:27.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:27.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T11:31:27.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:27.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:27.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:27.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:27.612 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:27.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:27.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:27.766 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:27.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:27.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:27.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:27.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:28.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:28.031 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:28.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:28.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:27 smithi179 ceph-mon[37907]: pgmap v525: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-24T11:31:28.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:27 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:31:28.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:27 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:31:28.045 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:27 smithi179 ceph-mon[29608]: pgmap v525: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-24T11:31:28.046 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:27 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:31:28.046 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:27 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:31:28.133 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:28.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:27 smithi181 ceph-mon[33998]: pgmap v525: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-24T11:31:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:27 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:31:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:27 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:31:28.186 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:28.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:28.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:28.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:28.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:28.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:28.469 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:28.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:28.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:28.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:28.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:28.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:28.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:28.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:28.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:28.881 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:28.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:28.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:29.026 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:29.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:29.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:28 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:31:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:28 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:31:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:28 smithi181 ceph-mon[33998]: 9.b scrub starts 2024-03-24T11:31:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:28 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:31:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:28 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:31:29.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:31:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:31:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[29608]: 9.b scrub starts 2024-03-24T11:31:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:31:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:31:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:31:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:31:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[37907]: 9.b scrub starts 2024-03-24T11:31:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:31:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:28 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:31:29.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.303 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:29.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:29.404 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.443 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:29.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:29.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.589 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:29.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:29.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:29.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:29.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.861 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:29.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[29608]: pgmap v526: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-24T11:31:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[37907]: pgmap v526: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:29 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:31:29.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:29.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:29.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:30.081 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:31:30.081 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:31:30.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:30.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:30.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:29 smithi181 ceph-mon[33998]: pgmap v526: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-24T11:31:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:29 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:31:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:29 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:31:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:29 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:31:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:29 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:31:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:29 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:31:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:29 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:31:30.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:30.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:30.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:30.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:30.411 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:30.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:30.513 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:30.551 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:30.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:30.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:30.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:30.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:30.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:30.826 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:30.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:30.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:31:30.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:31.007 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:31.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[29608]: 9.b scrub ok 2024-03-24T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[37907]: 9.b scrub ok 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:31:31.036 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:30 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:30 smithi181 ceph-mon[33998]: 9.b scrub ok 2024-03-24T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:30 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:30 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:30 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:30 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:30 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:30 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:31:31.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:31.147 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:31.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:31.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:31.289 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:31.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:31.391 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:31.424 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:31.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:31.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:31.549 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:31.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:31:31.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:31.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:31.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:31.759 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:31.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:31.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:31.916 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:31.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:32.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:32.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:31 smithi179 ceph-mon[29608]: pgmap v527: 168 pgs: 168 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:31:32.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:31 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:31:32.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:31 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:31:32.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:31 smithi179 ceph-mon[37907]: pgmap v527: 168 pgs: 168 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:31:32.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:31 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:31:32.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:31 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:31:32.058 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:32.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:31 smithi181 ceph-mon[33998]: pgmap v527: 168 pgs: 168 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:31:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:31 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:31:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:31 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:31:32.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:32.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:32.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:32.304 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:32.349 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:32.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:32.375 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:32.450 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:32.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:32.492 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:32.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:32.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:32.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:32.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:32.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:32.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:32.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:32.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:32.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[29608]: 4.0 scrub starts 2024-03-24T11:31:32.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[29608]: 4.0 scrub ok 2024-03-24T11:31:32.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[29608]: 1.8 deep-scrub starts 2024-03-24T11:31:32.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[29608]: 1.8 deep-scrub ok 2024-03-24T11:31:32.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[29608]: 9.e scrub starts 2024-03-24T11:31:32.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:31:32.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:31:32.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[37907]: 4.0 scrub starts 2024-03-24T11:31:32.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[37907]: 4.0 scrub ok 2024-03-24T11:31:32.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[37907]: 1.8 deep-scrub starts 2024-03-24T11:31:32.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[37907]: 1.8 deep-scrub ok 2024-03-24T11:31:32.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[37907]: 9.e scrub starts 2024-03-24T11:31:32.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:31:32.893 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:32 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:31:32.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:32.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:33.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:33.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:33.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:33.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:32 smithi181 ceph-mon[33998]: 4.0 scrub starts 2024-03-24T11:31:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:32 smithi181 ceph-mon[33998]: 4.0 scrub ok 2024-03-24T11:31:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:32 smithi181 ceph-mon[33998]: 1.8 deep-scrub starts 2024-03-24T11:31:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:32 smithi181 ceph-mon[33998]: 1.8 deep-scrub ok 2024-03-24T11:31:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:32 smithi181 ceph-mon[33998]: 9.e scrub starts 2024-03-24T11:31:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:32 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:31:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:32 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:31:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:33.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:33.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:33.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:33.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:33.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:33.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:33.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:33.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:33.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:33.809 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.851 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:33.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:33.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:33.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:33.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:34.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:34.126 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:34.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:34.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: pgmap v528: 168 pgs: 168 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: 9.e scrub ok 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: pgmap v528: 168 pgs: 168 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:31:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: 9.e scrub ok 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: pgmap v528: 168 pgs: 168 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: 9.e scrub ok 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:31:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:31:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:31:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:31:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:34.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:34.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:34.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:34.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:34.407 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:34.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:34.508 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:34.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:34.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:34.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:34.676 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:34.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:34.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:34.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:34.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:34.911 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:34.936 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:34.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:35.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:35.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:34 smithi179 ceph-mon[29608]: 1.e scrub starts 2024-03-24T11:31:35.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:34 smithi179 ceph-mon[29608]: 1.e scrub ok 2024-03-24T11:31:35.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:35.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:34 smithi179 ceph-mon[37907]: 1.e scrub starts 2024-03-24T11:31:35.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:34 smithi179 ceph-mon[37907]: 1.e scrub ok 2024-03-24T11:31:35.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:35.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:35.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:35.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:34 smithi181 ceph-mon[33998]: 1.e scrub starts 2024-03-24T11:31:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:34 smithi181 ceph-mon[33998]: 1.e scrub ok 2024-03-24T11:31:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:35.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:35.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:35.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:35.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:35.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:35.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:35.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:35.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:35.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:31:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:31:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:31:35.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:35.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:35.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:35.630 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:35.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:35.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:35.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:35.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:35.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:35.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:36.009 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:36.071 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:36.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:36.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:35 smithi179 ceph-mon[29608]: pgmap v529: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-03-24T11:31:36.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:35 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:31:36.113 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:35 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:31:36.114 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:35 smithi179 ceph-mon[37907]: pgmap v529: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-03-24T11:31:36.114 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:35 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:31:36.114 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:35 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:31:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:35 smithi181 ceph-mon[33998]: pgmap v529: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-03-24T11:31:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:35 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:31:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:35 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:31:36.172 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:36.212 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:36.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:36.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:36.345 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:36.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:36.351 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:31:36.398 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:31:36.436 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:31:36.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:36.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:36.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:36.481 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:31:36.522 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:31:36.563 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:31:36.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:36.602 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:31:36.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:36.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:36.645 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:31:36.650 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:31:36.650 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:31:36.650 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:31:36.650 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:31:36.650 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:31:36.651 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:31:36.651 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:31:36.651 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:31:36.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:36.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:36.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:36.848 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:36.887 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:36.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[29608]: 9.5 scrub starts 2024-03-24T11:31:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:31:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:31:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1046958581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[37907]: 9.5 scrub starts 2024-03-24T11:31:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:31:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:31:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1046958581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:36.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:37.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:37.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:36 smithi181 ceph-mon[33998]: 9.5 scrub starts 2024-03-24T11:31:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:36 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:31:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:36 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:31:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1046958581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:37.162 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:37.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:37.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:37.399 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:37.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:37.524 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:37.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:37.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:37.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:37.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:37.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:37.862 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:37.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:37.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:37.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:38.099 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[29608]: pgmap v530: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[29608]: 9.5 scrub ok 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[37907]: pgmap v530: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:31:38.100 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:37 smithi179 ceph-mon[37907]: 9.5 scrub ok 2024-03-24T11:31:38.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:38.133 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:38.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:38.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:37 smithi181 ceph-mon[33998]: pgmap v530: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:31:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:37 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:31:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:37 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:31:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:37 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:31:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:37 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:31:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:37 smithi181 ceph-mon[33998]: 9.5 scrub ok 2024-03-24T11:31:38.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:38.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:38.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:38.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:38.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:38.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:38.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:38.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:38.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:38.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:38.676 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:38.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:38.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:38.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:38.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:31:38.885 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:38 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:31:38.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:38.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:38.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:39.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:39.075 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:39.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:39.111 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:31:39.112 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:31:39.112 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:31:39.112 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:31:39.112 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:31:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:38 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:31:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:38 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:31:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:38 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:31:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:38 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:31:39.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:39.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:39.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:39.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:39.344 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:39.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:39.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:39.480 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:39.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:39.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:31:39.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:39.633 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:39.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:39.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:39.776 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:39.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:39.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:39.925 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:39.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:40.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:40.060 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:40.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:39 smithi181 ceph-mon[33998]: pgmap v531: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:31:40.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:40.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:40.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:40.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:39 smithi179 ceph-mon[29608]: pgmap v531: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:31:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:39 smithi179 ceph-mon[37907]: pgmap v531: 168 pgs: 168 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:31:40.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:40.331 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:40.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:40.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:40.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:40.478 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:40.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:40.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:40.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:40.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:40.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:40.749 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:40.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:40.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:31:40.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:40.885 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:40.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:40.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.018 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:41.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:41.033 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:40 smithi179 ceph-mon[29608]: 9.f scrub starts 2024-03-24T11:31:41.033 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:40 smithi179 ceph-mon[37907]: 9.f scrub starts 2024-03-24T11:31:41.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:41.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:40 smithi181 ceph-mon[33998]: 9.f scrub starts 2024-03-24T11:31:41.166 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:41.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:41.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:41.440 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:41.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:41.571 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:41.586 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:41.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:41.703 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:41.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:41.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:41.860 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T11:31:41.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:41.983 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:41.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:42.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:42.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:42.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:41 smithi181 ceph-mon[33998]: pgmap v532: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T11:31:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:41 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:31:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:41 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:31:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:41 smithi181 ceph-mon[33998]: 9.f scrub ok 2024-03-24T11:31:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:41 smithi181 ceph-mon[33998]: 9.9 deep-scrub starts 2024-03-24T11:31:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3635129987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:31:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:41 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:31:42.205 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[29608]: pgmap v532: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T11:31:42.206 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:31:42.206 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:31:42.206 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[29608]: 9.f scrub ok 2024-03-24T11:31:42.206 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[29608]: 9.9 deep-scrub starts 2024-03-24T11:31:42.206 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3635129987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:31:42.206 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:31:42.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[37907]: pgmap v532: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T11:31:42.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:31:42.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:31:42.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[37907]: 9.f scrub ok 2024-03-24T11:31:42.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[37907]: 9.9 deep-scrub starts 2024-03-24T11:31:42.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3635129987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:31:42.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:41 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:31:42.218 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:42.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:42.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:42.353 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:42.393 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:42.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:42.495 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:42.530 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:42.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:42.542 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:42.631 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:42.669 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:42.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:42.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:42.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:42.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:42.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:42.854 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:31:42.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:42.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:42.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:43.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:43.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:43.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 4.19 scrub starts 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 4.19 scrub ok 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 6.1d deep-scrub starts 2024-03-24T11:31:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:31:43.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T11:31:43.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 6.1d deep-scrub ok 2024-03-24T11:31:43.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1787855827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:31:43.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 1.7 deep-scrub starts 2024-03-24T11:31:43.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:42 smithi181 ceph-mon[33998]: 1.7 deep-scrub ok 2024-03-24T11:31:43.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 4.19 scrub starts 2024-03-24T11:31:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 4.19 scrub ok 2024-03-24T11:31:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 6.1d deep-scrub starts 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 6.1d deep-scrub ok 2024-03-24T11:31:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1787855827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:31:43.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 1.7 deep-scrub starts 2024-03-24T11:31:43.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[29608]: 1.7 deep-scrub ok 2024-03-24T11:31:43.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 4.19 scrub starts 2024-03-24T11:31:43.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 4.19 scrub ok 2024-03-24T11:31:43.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:31:43.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 6.1d deep-scrub starts 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: osdmap e249: 8 total, 8 up, 8 in 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 6.1d deep-scrub ok 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1787855827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 1.7 deep-scrub starts 2024-03-24T11:31:43.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:42 smithi179 ceph-mon[37907]: 1.7 deep-scrub ok 2024-03-24T11:31:43.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:43.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:43.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:43.364 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:43.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:43.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-24T11:31:43.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:43.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:43.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:43.600 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:43.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:43.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:43.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:43.783 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:43.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:43.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:43.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:43.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[29608]: pgmap v534: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-24T11:31:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:31:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:31:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:31:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1787855827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:31:43.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[29608]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T11:31:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[37907]: pgmap v534: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-24T11:31:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:31:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:31:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:31:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1787855827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:31:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:43 smithi179 ceph-mon[37907]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T11:31:44.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:44.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:43 smithi181 ceph-mon[33998]: pgmap v534: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-24T11:31:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:43 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:31:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:43 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:31:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:43 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:31:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:43 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1787855827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:31:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:43 smithi181 ceph-mon[33998]: osdmap e250: 8 total, 8 up, 8 in 2024-03-24T11:31:44.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:31:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:31:44.155 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:44.190 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:44.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:44.326 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:44.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:44.473 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:44.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:44.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:44.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:44.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:44.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:44.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:44.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:44.993 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:44.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:45.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:45.101 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:45.102 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:31:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:31:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:31:45.102 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:45.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:45.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:45.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:45.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:45.270 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:45.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:45.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:45.407 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:45.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:45.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:45.543 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:45.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:45.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:45.676 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:45.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:45.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:45.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:45.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:45.955 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:45 smithi179 ceph-mon[29608]: pgmap v536: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:31:45.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:45 smithi179 ceph-mon[37907]: pgmap v536: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:31:45.958 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:45.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:46.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:46.095 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:46.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:45 smithi181 ceph-mon[33998]: pgmap v536: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:31:46.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:46.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:46.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:46.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:46.366 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:46.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:46.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:46.507 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:46.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:46.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:46.638 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:46.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:46.660 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:31:46.702 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:31:46.740 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:31:46.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:46.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:46.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:46.777 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:31:46.819 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:31:46.868 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:46.871 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:31:46.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:46.916 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:31:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:46.937 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:46.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:46.960 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:31:46.965 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:31:47.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:47.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:47.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:47.075 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1515242568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:47.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1515242568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:47.104 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:31:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1515242568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:31:47.173 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:47.182 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:31:47.182 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:31:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:47.232 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:47.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:47.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:47.383 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:47.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:47.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:47.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:47.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:47.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:47.660 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:47.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:47.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:47.801 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:47.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:47.903 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:47.936 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:47.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:47.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[29608]: pgmap v537: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-24T11:31:47.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[29608]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[37907]: pgmap v537: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[37907]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T11:31:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:48.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:48.076 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:48.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:47 smithi181 ceph-mon[33998]: pgmap v537: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-24T11:31:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:47 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:31:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:47 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:31:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:47 smithi181 ceph-mon[33998]: osdmap e251: 8 total, 8 up, 8 in 2024-03-24T11:31:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:31:48.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:48.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:48.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:48.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:31:48.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:48.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:48.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:48.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:48.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:48.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:48.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:48.658 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:48.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:48.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:48.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:48.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:48.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:48.927 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:48.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:49.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:49.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:49.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:49.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:49.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:49.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:49.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:49.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:49.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:49.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:49.460 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:49.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:49.562 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:49.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:49.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:49 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:49 smithi181 ceph-mon[33998]: pgmap v539: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 795 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:31:49.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:49 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:49.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:49.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:49 smithi179 ceph-mon[29608]: pgmap v539: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 795 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:31:49.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:49 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:49.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:31:49.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:49 smithi179 ceph-mon[37907]: pgmap v539: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 795 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:31:49.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:49.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:49.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:49.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:49.871 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:49.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:49.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:50.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:50.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:50.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:50.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:50.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:50.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:50.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:50.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:50.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:50.545 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:50.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:50.685 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:50.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:50.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:50.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:50.953 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:50.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:51.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:51.100 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:51.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:51.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:51.244 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:51.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:51.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:51.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:51.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:51.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:51 smithi179 ceph-mon[29608]: pgmap v540: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-03-24T11:31:51.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:51 smithi179 ceph-mon[37907]: pgmap v540: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-03-24T11:31:51.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:51.526 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:51.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:51.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:51 smithi181 ceph-mon[33998]: pgmap v540: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-03-24T11:31:51.664 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:51.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:51.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:51.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:51.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:51.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:51.938 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:51.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:52.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:52.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:52.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:52.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:52.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:52.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:52.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:52.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:52.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:52.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:52.481 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:52.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:52.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:52.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:52 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:52.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:52 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:52.639 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:52 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:52.640 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:52 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:52.640 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:52 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:52.640 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:52 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:52.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:52.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:52.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:52 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:31:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:52 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:31:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:52 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:31:52.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:52.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:52.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:52.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:52.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:52.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:53.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:53.047 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:53.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:53.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:53.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:53.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:53.183 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:53.190 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:31:53.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:53.327 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:53.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:53.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:53.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:53.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:53.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:53.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:53.604 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:53 smithi179 ceph-mon[29608]: pgmap v541: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-24T11:31:53.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:53 smithi179 ceph-mon[37907]: pgmap v541: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-24T11:31:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:53.608 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:53.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:53.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:53 smithi181 ceph-mon[33998]: pgmap v541: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-24T11:31:53.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:53.753 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:53.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:53.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:53.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:53.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:53.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:54.025 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:54.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:54.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:31:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:31:54.164 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:54.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:54.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:54.382 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.4 to 1 (10000) 2024-03-24T11:31:54.396 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.439 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:54.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:54.540 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:54.573 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:54.634 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1259994905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:31:54.634 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:54.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1259994905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:31:54.635 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:54.642 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:31:54.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1259994905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:31:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:31:54.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:54.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:54.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:54.840 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:54.942 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:54.981 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:54.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:55.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:55.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:55.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:55.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:55.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:55.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:55.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:55.378 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:31:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:31:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:31:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:55.397 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:55.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:55.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T11:31:55.498 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:55.541 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:55.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:55.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:55.642 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:55 smithi181 ceph-mon[33998]: pgmap v542: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:31:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:55 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1259994905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:31:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:55 smithi181 ceph-mon[33998]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T11:31:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:55.688 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:55.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:55 smithi179 ceph-mon[29608]: pgmap v542: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:31:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:55 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1259994905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:31:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:55 smithi179 ceph-mon[29608]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T11:31:55.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:55 smithi179 ceph-mon[37907]: pgmap v542: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T11:31:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:55 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1259994905' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:31:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:55 smithi179 ceph-mon[37907]: osdmap e252: 8 total, 8 up, 8 in 2024-03-24T11:31:55.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:55.824 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:55.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:55.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:55.983 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:55.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:56.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:56.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:56.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:56.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:56.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:56.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:56.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:56.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:56.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:56.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:56.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:56.525 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:56.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:56.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:56.652 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:56.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:56.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:56.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:56.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:56.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:56.930 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:56.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:57.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:57.066 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:57.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:57.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:57.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:57.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:57.299 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:57.337 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:57.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:57.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:57.475 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:57.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:57.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:57.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:57.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:57.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:57 smithi181 ceph-mon[33998]: pgmap v544: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:31:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:57 smithi179 ceph-mon[29608]: pgmap v544: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:31:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:57 smithi179 ceph-mon[37907]: pgmap v544: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T11:31:57.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:57.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:57.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:57.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:57.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:57.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:57.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:57.988 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:31:58.010 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:31:58.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:58.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:58.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:31:58.284 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:31:58.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:58.410 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:31:58.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:58.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:58.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:31:58.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:31:58.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.821 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:58.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:58.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:58.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:58.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:59.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:31:59.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:59.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:31:59.192 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:31:59.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:59.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:31:59.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:59.374 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:59.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:59.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:59.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:59.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:59.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:31:59.645 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:31:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:59.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:31:59.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:31:59 smithi181 ceph-mon[33998]: pgmap v545: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:31:59.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:31:59 smithi179 ceph-mon[29608]: pgmap v545: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:31:59.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:31:59 smithi179 ceph-mon[37907]: pgmap v545: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:31:59.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:31:59.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:59.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:31:59.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:31:59.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:31:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:31:59.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:00.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:00.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:00.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:00.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:00.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:00.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:00.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:00.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:00.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:00.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:00.627 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:00.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:00.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:00 smithi181 ceph-mon[33998]: 9.9 deep-scrub ok 2024-03-24T11:32:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:00 smithi179 ceph-mon[29608]: 9.9 deep-scrub ok 2024-03-24T11:32:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:00 smithi179 ceph-mon[37907]: 9.9 deep-scrub ok 2024-03-24T11:32:00.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:00.763 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:00.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:00.865 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:00.888 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:00.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:00.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:01.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:01.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.165 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:01.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:01.266 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.308 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:01.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:01.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:01.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:01.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:01.570 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:01.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:01 smithi181 ceph-mon[33998]: pgmap v546: 168 pgs: 168 active+clean; 2.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:32:01.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:01 smithi179 ceph-mon[29608]: pgmap v546: 168 pgs: 168 active+clean; 2.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:32:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:01 smithi179 ceph-mon[37907]: pgmap v546: 168 pgs: 168 active+clean; 2.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:32:01.716 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:01.817 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:01.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:01.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:01.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:01.994 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:01.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:02.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:02.131 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:02.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:02.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:02.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:02.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:02.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:02.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:02.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:02.498 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:02.502 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:02.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:02.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:02.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:02.697 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:02.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:02.798 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:02.842 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:02.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:02.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:03.015 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:03.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:03.144 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:03.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:03.284 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:03.298 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:32:03.298 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:32:03.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:03.416 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:03.433 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:32:03.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:03.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:03.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:03 smithi181 ceph-mon[33998]: pgmap v547: 168 pgs: 168 active+clean; 2.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:32:03.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:03 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3349604787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:03.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:03.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:03.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:03 smithi179 ceph-mon[37907]: pgmap v547: 168 pgs: 168 active+clean; 2.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:32:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3349604787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:03 smithi179 ceph-mon[29608]: pgmap v547: 168 pgs: 168 active+clean; 2.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T11:32:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:03 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3349604787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:03.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:03.827 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:03.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:03.962 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:03.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:04.064 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:04.103 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:04.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:04.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:32:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:32:04.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:04.243 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:04.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:04.345 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:04.384 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:04.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:04.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:04.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3349604787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:04.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[29608]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T11:32:04.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:04.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:04.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3349604787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:04.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[37907]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T11:32:04.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:04.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:04.516 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:04.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:04.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:04.656 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:04.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:04.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:04.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:04.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:04.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:04 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3349604787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:04 smithi181 ceph-mon[33998]: osdmap e253: 8 total, 8 up, 8 in 2024-03-24T11:32:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:04.931 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:04.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:05.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:05.070 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:05.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:05.172 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:05.209 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:05.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:05.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:05.348 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:05.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:05.422 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:32:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:32:05.451 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:05.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:05.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:05.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:05.628 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:05.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[29608]: pgmap v549: 168 pgs: 168 active+clean; 1.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-24T11:32:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:32:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:32:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[29608]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T11:32:05.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[37907]: pgmap v549: 168 pgs: 168 active+clean; 1.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-24T11:32:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:32:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:32:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:05 smithi179 ceph-mon[37907]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T11:32:05.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:05.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:05.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:05.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:05.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:05.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:05.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:05 smithi181 ceph-mon[33998]: pgmap v549: 168 pgs: 168 active+clean; 1.7 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-24T11:32:05.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:05 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:32:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:05 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:32:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:05 smithi181 ceph-mon[33998]: osdmap e254: 8 total, 8 up, 8 in 2024-03-24T11:32:05.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:06.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:06.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:06.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:06.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:06.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:06.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:06.433 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:06.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:06.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:32:06.574 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:06.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:06.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:06 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:32:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:06 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:32:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:06 smithi179 ceph-mon[29608]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T11:32:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:06 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:32:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:06 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:32:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:06 smithi179 ceph-mon[37907]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T11:32:06.807 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:06.847 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:06 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:32:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:06 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:32:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:06 smithi181 ceph-mon[33998]: osdmap e255: 8 total, 8 up, 8 in 2024-03-24T11:32:06.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:06.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:06.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:06.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:07.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:07.147 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:07.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:07.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:07.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:07.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:07.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:07.423 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:07.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:07.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:07.563 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:07.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:07.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:07.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:07 smithi179 ceph-mon[29608]: pgmap v552: 168 pgs: 168 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:32:07.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:07 smithi179 ceph-mon[37907]: pgmap v552: 168 pgs: 168 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:32:07.724 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:07.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:07.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:07.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:07.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:07 smithi181 ceph-mon[33998]: pgmap v552: 168 pgs: 168 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T11:32:07.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:07.996 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:07.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:08.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:08.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:32:08.138 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:08.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:08.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:08.270 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:08.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:08.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:08.429 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:08.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:08.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:08.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:08.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:08.668 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:08.678 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:08.730 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:08.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:08.793 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:32:08.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:08.837 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:32:08.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:08.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:08.877 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:32:08.923 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:32:08.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:08.967 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:32:09.013 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:32:09.013 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:09.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:09.052 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:32:09.088 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:32:09.092 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:32:09.092 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:32:09.092 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:32:09.093 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:32:09.093 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:32:09.093 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:32:09.093 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:32:09.093 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:32:09.115 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:09.160 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:09.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.297 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:09.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:09.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:09.434 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:09.469 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T11:32:09.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:09.568 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:09.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[29608]: pgmap v553: 168 pgs: 168 active+clean; 1.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:32:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2164456740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3877298282' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:32:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:32:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[37907]: pgmap v553: 168 pgs: 168 active+clean; 1.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:32:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2164456740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3877298282' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:32:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:32:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.711 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:09.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:09.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:09.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:09.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:09 smithi181 ceph-mon[33998]: pgmap v553: 168 pgs: 168 active+clean; 1.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-24T11:32:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2164456740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3877298282' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:32:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:32:09.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:09.984 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:09.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:10.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:10.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:10.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:10.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:10.257 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:10.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:10.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T11:32:10.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:10.405 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:10.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:10.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:10.541 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:10.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:10.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:10.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:10.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:10 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:32:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:10 smithi179 ceph-mon[29608]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T11:32:10.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:10 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:32:10.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:10 smithi179 ceph-mon[37907]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T11:32:10.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:10.818 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:10.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:10.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:10 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:32:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:10 smithi181 ceph-mon[33998]: osdmap e256: 8 total, 8 up, 8 in 2024-03-24T11:32:10.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:10.969 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:10.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:11.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:11.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:11.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:11.205 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:11.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:11.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:11.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:11.366 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:11.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:11.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:11.497 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:11.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:11.599 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:11.637 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:11.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:11.738 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:11.766 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:11.786 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:11.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:11.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:11 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:32:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:11 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:32:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:11 smithi181 ceph-mon[33998]: pgmap v555: 168 pgs: 168 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:32:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:11 smithi181 ceph-mon[33998]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T11:32:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:11 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:32:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:11 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:32:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:11.920 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:11.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:32:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:32:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[29608]: pgmap v555: 168 pgs: 168 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:32:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[29608]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T11:32:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:32:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:32:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:32:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:32:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[37907]: pgmap v555: 168 pgs: 168 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:32:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[37907]: osdmap e257: 8 total, 8 up, 8 in 2024-03-24T11:32:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:32:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:11 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:32:12.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:12.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:12.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:12.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:12.192 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:12.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:12.294 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:12.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:12.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:12.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:12.478 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:12.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:12.580 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:12.621 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:12.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:12.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:12.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:12.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:12.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:12.800 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:12 smithi179 ceph-mon[29608]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T11:32:12.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:12 smithi179 ceph-mon[37907]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T11:32:12.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:12 smithi181 ceph-mon[33998]: osdmap e258: 8 total, 8 up, 8 in 2024-03-24T11:32:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:12.919 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:12.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:13.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:13.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:13.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:13.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:13.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:13.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:13.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:13.332 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:13.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:13.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:13.466 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:13.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:13.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:13.545 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:32:13.546 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:32:13.546 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:32:13.546 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:32:13.546 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:32:13.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:13.636 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:13.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[29608]: pgmap v558: 168 pgs: 168 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:32:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:32:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:32:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[29608]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T11:32:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[37907]: pgmap v558: 168 pgs: 168 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:32:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:32:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:32:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:13 smithi179 ceph-mon[37907]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T11:32:13.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:13.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:13.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:13.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:13 smithi181 ceph-mon[33998]: pgmap v558: 168 pgs: 168 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:32:13.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:13 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:32:13.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:13 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:32:13.811 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:13 smithi181 ceph-mon[33998]: osdmap e259: 8 total, 8 up, 8 in 2024-03-24T11:32:13.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:13.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:13.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:14.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:14.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:14.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:14.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:14.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:32:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:32:14.180 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:14.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:14.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:14.318 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:14.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:14.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:14.455 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:14.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:14.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:32:14.558 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:14.633 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:14.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:14.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:14.765 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:14.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:14.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:32:14.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:14.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:14.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:14.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:14.939 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:14.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:15.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:15.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:15.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:15.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:15.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:32:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:32:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:32:15.223 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:15.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:15.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:15.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:15.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:15.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:15.500 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:15.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:15.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:15.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:15.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:15.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:15.747 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:15 smithi179 ceph-mon[29608]: pgmap v560: 168 pgs: 168 active+clean; 1.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 697 op/s 2024-03-24T11:32:15.747 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:15 smithi179 ceph-mon[37907]: pgmap v560: 168 pgs: 168 active+clean; 1.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 697 op/s 2024-03-24T11:32:15.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:15.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:15.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:15 smithi181 ceph-mon[33998]: pgmap v560: 168 pgs: 168 active+clean; 1.0 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 697 op/s 2024-03-24T11:32:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:15.925 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:15.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:16.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:16.061 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:16.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:16.163 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:16.213 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:16.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:16.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:16.348 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:16.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:16.450 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:16.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:16.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:16.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:16.628 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:16.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:16.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:16.763 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:16.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:16.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:16.900 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:16.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:17.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:17.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:17.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:17.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:17.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:17.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:17.168 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:17.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:17.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:17.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:17.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:17.454 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:17.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:17.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:17.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:17.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:17.689 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:17.736 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:17.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:17.838 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:17.839 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[29608]: pgmap v561: 168 pgs: 168 active+clean; 822 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:32:17.839 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:32:17.840 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:32:17.840 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[29608]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T11:32:17.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[37907]: pgmap v561: 168 pgs: 168 active+clean; 822 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:32:17.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:32:17.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:32:17.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:17 smithi179 ceph-mon[37907]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T11:32:17.882 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:17.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:17 smithi181 ceph-mon[33998]: pgmap v561: 168 pgs: 168 active+clean; 822 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T11:32:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:32:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:32:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:17 smithi181 ceph-mon[33998]: osdmap e260: 8 total, 8 up, 8 in 2024-03-24T11:32:17.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:18.015 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:18.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:18.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:18.254 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:18.278 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:18.380 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:18.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:18.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:18.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.557 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:18.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:18.564 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:32:18.581 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T11:32:18.659 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:18.697 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:18.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:18.831 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:18.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:18 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/846691077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:32:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:18 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2524240962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:32:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:18 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:32:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:18.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/846691077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:32:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2524240962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:32:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:32:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/846691077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:32:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2524240962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:32:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:32:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:18.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:18.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:19.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:19.101 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:19.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:19.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:19.231 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:19.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:19.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:19.368 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:19.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:19.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:19.506 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:19.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:19.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:19.645 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:19.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:19.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:19.788 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:19.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:19.889 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:19 smithi181 ceph-mon[33998]: pgmap v563: 168 pgs: 168 active+clean; 796 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 753 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T11:32:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:19 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/846691077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:32:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:19 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:32:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:19 smithi181 ceph-mon[33998]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T11:32:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:19 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:32:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:19 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[29608]: pgmap v563: 168 pgs: 168 active+clean; 796 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 753 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/846691077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[29608]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[37907]: pgmap v563: 168 pgs: 168 active+clean; 796 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 753 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/846691077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:32:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:32:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[37907]: osdmap e261: 8 total, 8 up, 8 in 2024-03-24T11:32:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:32:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:19 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:32:19.925 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:19.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:20.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:20.064 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:20.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:20.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:20.205 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:20.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:20.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:20.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:20.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:20.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:20.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:20.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:20.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:20.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:20.615 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:20.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:20.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:20 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:32:20.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:20 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:32:20.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:20 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:32:20.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:20 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:32:20.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:20.750 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:20.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:20.852 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:20.888 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:20.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:20.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:20 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:32:20.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:20 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:32:20.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:21.028 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:21.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:21.130 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:21.169 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:21.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:21.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:21.307 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:21.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:21.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:21.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-24T11:32:21.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:21.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:21.451 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:21.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:21.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:21.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:21.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:21.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:21 smithi179 ceph-mon[29608]: pgmap v565: 168 pgs: 168 active+clean; 411 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-24T11:32:21.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:21 smithi179 ceph-mon[37907]: pgmap v565: 168 pgs: 168 active+clean; 411 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-24T11:32:21.727 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:21.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:21.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:21.867 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:21.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:21.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:21 smithi181 ceph-mon[33998]: pgmap v565: 168 pgs: 168 active+clean; 411 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-24T11:32:21.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:22.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:22.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:22.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:22.141 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:22.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:22.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:22.279 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:22.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:22.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:22.396 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:32:22.445 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:22.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:22.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:22.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:22.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:22.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:22.737 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:22.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:22.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:22.979 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:22.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:23.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:23.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:23.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:23.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:23.246 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:23.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:23.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:23.391 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:23.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:23.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:23.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:23.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:23.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:23.663 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:23.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:23.757 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:23.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:23.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:23.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[29608]: pgmap v566: 168 pgs: 168 active+clean; 411 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:32:23.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:32:23.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:32:23.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:32:23.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[37907]: pgmap v566: 168 pgs: 168 active+clean; 411 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:32:23.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:32:23.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:23 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:32:23.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:32:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:32:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:32:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:32:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:32:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-mon[33998]: pgmap v566: 168 pgs: 168 active+clean; 411 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:32:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:32:23.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:23 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:32:23.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:23.947 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:23.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:24.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:24.090 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:24.097 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:32:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:32:24.151 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Removed 133584 objects 2024-03-24T11:32:24.152 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Clean up completed and total clean up time :105.637 2024-03-24T11:32:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:32:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:32:24.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:32:24.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:32:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:32:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:32:24.270 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:32:24.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:24.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:24.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.467 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:24.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:24.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:24.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:24.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:24.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:24.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:24.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:24.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:32:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:32:24.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:24 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1633265407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:24.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:32:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:32:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T11:32:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:32:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:32:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1633265407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: osdmap e262: 8 total, 8 up, 8 in 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:32:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:24 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1633265407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:24.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:24.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:25.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:25.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:25.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:25.218 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:25.235 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:32:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:32:25.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:25.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:25.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:25.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:25.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:25.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:25.666 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:25.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:25.724 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:25.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:25.849 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:25 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:32:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:25 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:32:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:25 smithi181 ceph-mon[33998]: pgmap v568: 168 pgs: 168 active+clean; 211 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-24T11:32:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:25 smithi181 ceph-mon[33998]: 1.b scrub starts 2024-03-24T11:32:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:25 smithi181 ceph-mon[33998]: 1.b scrub ok 2024-03-24T11:32:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:25 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:32:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:25 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:32:25.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:32:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:32:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[29608]: pgmap v568: 168 pgs: 168 active+clean; 211 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-24T11:32:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[29608]: 1.b scrub starts 2024-03-24T11:32:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[29608]: 1.b scrub ok 2024-03-24T11:32:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:32:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:32:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:32:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:32:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[37907]: pgmap v568: 168 pgs: 168 active+clean; 211 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-24T11:32:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[37907]: 1.b scrub starts 2024-03-24T11:32:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[37907]: 1.b scrub ok 2024-03-24T11:32:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:32:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:25 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:32:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:25.976 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:25.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:26.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:26.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:26.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:26.328 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:26.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:26.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:26.480 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:26.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:26.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:26.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:26.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:26.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:32:26.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:32:26.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:32:26.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:32:26.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:32:26.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:32:26.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:32:26.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:26 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:32:26.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:26.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:26 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:32:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:26 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:32:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:26 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:32:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:26 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:32:26.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:26.987 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:26.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:27.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:27.113 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:27.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:27.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:27.238 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:27.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:27.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:27.368 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:27.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:27.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:27.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:27.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:27.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:27.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:27.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:27.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:27.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:27.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:27.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:27.876 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:27.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:27.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:27 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:32:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:27 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:32:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:27 smithi181 ceph-mon[33998]: pgmap v569: 168 pgs: 168 active+clean; 19 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-24T11:32:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:27 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:32:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:27 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:32:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:27 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:32:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:27 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:32:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:32:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:32:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[29608]: pgmap v569: 168 pgs: 168 active+clean; 19 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-24T11:32:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:32:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:32:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:32:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:32:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:32:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:32:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[37907]: pgmap v569: 168 pgs: 168 active+clean; 19 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-24T11:32:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:32:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:32:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:32:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:27 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:32:27.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:28.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:28.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:28.128 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:28.230 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:28.253 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:28.355 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:28.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:28.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:28.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:28.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:28.680 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:28.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:28.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:28.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:28.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 3.2 scrub starts 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[29608]: 3.2 scrub ok 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:32:28.897 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:32:28.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:32:28.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:32:28.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:32:28.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:32:28.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 3.2 scrub starts 2024-03-24T11:32:28.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:28 smithi179 ceph-mon[37907]: 3.2 scrub ok 2024-03-24T11:32:28.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:32:28.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:32:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:32:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:32:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:32:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:32:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 3.2 scrub starts 2024-03-24T11:32:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:28 smithi181 ceph-mon[33998]: 3.2 scrub ok 2024-03-24T11:32:28.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:28.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:28.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:29.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:29.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:29.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:29.184 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:29.286 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:29.309 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:29.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:29.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:29.454 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:32:29.454 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:32:29.538 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:29.562 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:29.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:29.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:29.789 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-24T11:32:29.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-24T11:32:29.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:29.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:29 smithi181 ceph-mon[33998]: pgmap v570: 168 pgs: 168 active+clean; 15 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T11:32:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:29 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:32:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:29 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:32:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:29 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:32:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:29 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:32:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:29 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1520463227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:32:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:29 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[29608]: pgmap v570: 168 pgs: 168 active+clean; 15 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1520463227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[37907]: pgmap v570: 168 pgs: 168 active+clean; 15 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T11:32:29.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:32:29.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:32:29.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:32:29.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:32:29.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1520463227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:32:29.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:29 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:32:29.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:29.940 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:29.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:30.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:30.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:30.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:30.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:30.212 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.7 to 1 (10000) 2024-03-24T11:32:30.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:30.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:30.430 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:30.454 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:30.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:30.579 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:30.681 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:30.704 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:30.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:30.830 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:30 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:32:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:30 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:32:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:30 smithi181 ceph-mon[33998]: 3.18 scrub starts 2024-03-24T11:32:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:30 smithi181 ceph-mon[33998]: 3.18 scrub ok 2024-03-24T11:32:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:30 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:32:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:30 smithi181 ceph-mon[33998]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T11:32:30.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:32:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:32:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[29608]: 3.18 scrub starts 2024-03-24T11:32:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[29608]: 3.18 scrub ok 2024-03-24T11:32:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:32:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[29608]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T11:32:30.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:32:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:32:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[37907]: 3.18 scrub starts 2024-03-24T11:32:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[37907]: 3.18 scrub ok 2024-03-24T11:32:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:32:30.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:30 smithi179 ceph-mon[37907]: osdmap e263: 8 total, 8 up, 8 in 2024-03-24T11:32:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:30.961 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:30.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:31.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:31.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:31.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:31.218 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:31.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:31.344 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:31.398 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T11:32:31.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:32:31.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:31.470 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:31.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:31.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:31.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[37907]: pgmap v572: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 822 op/s 2024-03-24T11:32:31.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:32:31.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:32:31.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:32:31.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:32:31.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[29608]: pgmap v572: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 822 op/s 2024-03-24T11:32:31.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:32:31.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:32:31.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:32:31.660 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:31 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:32:31.666 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:31.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:31.730 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:31.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:31.856 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:31.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:31 smithi181 ceph-mon[33998]: pgmap v572: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 822 op/s 2024-03-24T11:32:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:31 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:32:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:31 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:32:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:31 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:32:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:31 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:32:31.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:31.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:31.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:32.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:32.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:32.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:32.216 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:32.240 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:32.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:32.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:32.370 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:32.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:32.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:32.495 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:32.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:32.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:32.619 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:32:32.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:32.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:32.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:32.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:32.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:32.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:32.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:32 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:32:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:32 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:32:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:32 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:32:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:32 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:32:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:32 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3525226514' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:32:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:32:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:32:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:32:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3525226514' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:32:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:32:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:32:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:32:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:32 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3525226514' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:32.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:33.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:33.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:33.132 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:33.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:33.258 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:33.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:33.384 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:33.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:33.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:33.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:33.639 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:33.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:32:33.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: pgmap v573: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 524 op/s 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3525226514' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: 9.d scrub starts 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: 9.d scrub ok 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: 6.15 deep-scrub starts 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: 6.15 deep-scrub ok 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:33.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: pgmap v573: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 524 op/s 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3525226514' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: 9.d scrub starts 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: 9.d scrub ok 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: 6.15 deep-scrub starts 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: 6.15 deep-scrub ok 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:33.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:33.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:33.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:33.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:33.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:33.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:33.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: pgmap v573: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 524 op/s 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3525226514' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: osdmap e264: 8 total, 8 up, 8 in 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: 9.d scrub starts 2024-03-24T11:32:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: 9.d scrub ok 2024-03-24T11:32:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: 6.15 deep-scrub starts 2024-03-24T11:32:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: 6.15 deep-scrub ok 2024-03-24T11:32:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:33.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:33.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:34.023 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:34.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:34.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:34.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:34.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:34.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:34.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:34.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:34.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:34.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:32:34.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:34.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:34.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:34.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:34.879 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:34.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:34.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:34 smithi181 ceph-mon[33998]: 1.13 deep-scrub starts 2024-03-24T11:32:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:34 smithi181 ceph-mon[33998]: 1.13 deep-scrub ok 2024-03-24T11:32:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:34 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:32:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:34 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:32:34.911 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:34.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:34.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[29608]: 1.13 deep-scrub starts 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[29608]: 1.13 deep-scrub ok 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[37907]: 1.13 deep-scrub starts 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[37907]: 1.13 deep-scrub ok 2024-03-24T11:32:34.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:32:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:34 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:32:35.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:35.037 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:35.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:35.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:35.266 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.291 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:35.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:35.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:35.416 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:35.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:32:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:32:35.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.545 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:35.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:35.571 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:35.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:35.671 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:35.773 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:35.798 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[29608]: pgmap v575: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[29608]: 1.f deep-scrub starts 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[29608]: 1.f deep-scrub ok 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[37907]: pgmap v575: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-24T11:32:35.814 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:32:35.815 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:32:35.815 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[37907]: 1.f deep-scrub starts 2024-03-24T11:32:35.815 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[37907]: 1.f deep-scrub ok 2024-03-24T11:32:35.815 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:32:35.815 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:35 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:32:35.848 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:32:35.859 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:35.872 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:32:35.893 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:32:35.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:35 smithi181 ceph-mon[33998]: pgmap v575: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-24T11:32:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:35 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:32:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:35 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:32:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:35 smithi181 ceph-mon[33998]: 1.f deep-scrub starts 2024-03-24T11:32:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:35 smithi181 ceph-mon[33998]: 1.f deep-scrub ok 2024-03-24T11:32:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:35 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:32:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:35 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:32:35.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:32:35.923 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:35.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:35.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:35.932 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:32:35.953 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:32:35.973 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:32:35.993 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:32:35.994 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:32:35.994 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:32:35.994 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:32:35.994 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:32:35.995 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:32:35.995 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:32:35.995 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:32:35.995 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:32:36.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:36.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:36.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:36.177 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:36.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:36.303 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:36.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:36.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:36.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.556 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:36.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:36.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:36.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:36.762 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:32:36.763 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:32:36.763 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-03-24T11:32:36.763 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:32:36.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:36.808 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:36.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:32:36.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:32:36.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[37907]: 9.4 deep-scrub starts 2024-03-24T11:32:36.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[37907]: 9.4 deep-scrub ok 2024-03-24T11:32:36.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/227600131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:36.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:32:36.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:32:36.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[29608]: 9.4 deep-scrub starts 2024-03-24T11:32:36.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[29608]: 9.4 deep-scrub ok 2024-03-24T11:32:36.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/227600131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:36.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:36 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:32:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:36 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:32:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:36 smithi181 ceph-mon[33998]: 9.4 deep-scrub starts 2024-03-24T11:32:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:36 smithi181 ceph-mon[33998]: 9.4 deep-scrub ok 2024-03-24T11:32:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/227600131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:36.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:36.935 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:36.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:37.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:37.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:37.163 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:37.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:37.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:37.313 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:37.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:32:37.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:37.438 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:37.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:37.545 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:32:37.565 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:37.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:37.616 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:37.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:37.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:37.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:37.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:37.816 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:37.893 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: pgmap v576: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 30 op/s 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:32:37.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:37 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:32:37.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:37.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:37.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: pgmap v576: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 30 op/s 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:32:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: pgmap v576: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 30 op/s 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:32:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:37 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:32:38.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:38.067 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:38.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:38.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:38.294 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:38.318 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:38.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:38.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:38.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:38.569 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:38.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:38.695 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:38.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:38.820 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 9.0 scrub starts 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 9.0 scrub ok 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 9.6 deep-scrub starts 2024-03-24T11:32:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 9.6 deep-scrub ok 2024-03-24T11:32:38.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:32:38.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:38 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:32:38.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:32:38.922 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:38.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:32:38.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 9.0 scrub starts 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 9.0 scrub ok 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 9.6 deep-scrub starts 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 9.6 deep-scrub ok 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:32:38.940 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 9.0 scrub starts 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 9.0 scrub ok 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 9.6 deep-scrub starts 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 9.6 deep-scrub ok 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:32:38.941 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:38 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:32:38.964 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:38.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:39.066 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:39.089 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:39.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.217 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:39.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:39.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:39.343 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:39.374 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:39.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:39.469 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:39.559 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:32:39.560 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:32:39.560 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:32:39.560 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:32:39.560 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:32:39.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:39.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:39.696 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:39.721 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:39.786 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:39.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:39.846 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:39.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: pgmap v577: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-03-24T11:32:39.880 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:32:39.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:32:39.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:32:39.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:32:39.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:32:39.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:32:39.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 9.1 scrub starts 2024-03-24T11:32:39.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:39 smithi181 ceph-mon[33998]: 9.1 scrub ok 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: pgmap v577: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:32:39.891 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 9.1 scrub starts 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[29608]: 9.1 scrub ok 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: pgmap v577: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 9.1 scrub starts 2024-03-24T11:32:39.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:39 smithi179 ceph-mon[37907]: 9.1 scrub ok 2024-03-24T11:32:39.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:39.973 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:39.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:40.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:40.096 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:40.110 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:32:40.113 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:32:40.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:32:40.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:40.222 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:40.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:40.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:40.431 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:40.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:40.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:40.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:40.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:40.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:40.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:40.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 5.1 deep-scrub starts 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 5.1 deep-scrub ok 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 1.14 scrub starts 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[37907]: 1.14 scrub ok 2024-03-24T11:32:40.755 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 5.1 deep-scrub starts 2024-03-24T11:32:40.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 5.1 deep-scrub ok 2024-03-24T11:32:40.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:32:40.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:32:40.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:32:40.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:32:40.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 1.14 scrub starts 2024-03-24T11:32:40.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:40 smithi179 ceph-mon[29608]: 1.14 scrub ok 2024-03-24T11:32:40.784 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2024-03-24T11:32:38.300289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1276,"num_read_kb":1028,"num_write":267892,"num_write_kb":8549422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":131622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76656,"ondisk_log_size":76656,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3380800,"kb_used_data":7472,"kb_used_omap":12,"kb_used_meta":3372979,"kb_avail":746580416,"statfs":{"total":767960285184,"available":764498345984,"internally_reserved":0,"allocated":7651328,"data_stored":5250964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3453931057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.093579"},"pg_stats":[{"pgid":"9.1","version":"262'16752","reported_seq":16869,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617251+0000","last_change":"2024-03-24T11:28:58.838691+0000","last_active":"2024-03-24T11:32:32.617251+0000","last_peered":"2024-03-24T11:32:32.617251+0000","last_clean":"2024-03-24T11:32:32.617251+0000","last_became_active":"2024-03-24T11:28:58.838316+0000","last_became_peered":"2024-03-24T11:28:58.838316+0000","last_unstale":"2024-03-24T11:32:32.617251+0000","last_undegraded":"2024-03-24T11:32:32.617251+0000","last_fullsized":"2024-03-24T11:32:32.617251+0000","mapping_epoch":211,"log_start":"248'12600","ondisk_log_start":"248'12600","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:28:57.428807+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":536064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"213'20","reported_seq":487,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:36.799160+0000","last_change":"2024-03-24T11:32:36.799202+0000","last_active":"2024-03-24T11:32:36.799160+0000","last_peered":"2024-03-24T11:32:36.799160+0000","last_clean":"2024-03-24T11:32:36.799160+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:32:36.799160+0000","last_undegraded":"2024-03-24T11:32:36.799160+0000","last_fullsized":"2024-03-24T11:32:36.799160+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:36.799084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.799084+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:06.179015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":445,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617799+0000","last_change":"2024-03-24T11:29:27.048143+0000","last_active":"2024-03-24T11:32:32.617799+0000","last_peered":"2024-03-24T11:32:32.617799+0000","last_clean":"2024-03-24T11:32:32.617799+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:32:32.617799+0000","last_undegraded":"2024-03-24T11:32:32.617799+0000","last_fullsized":"2024-03-24T11:32:32.617799+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:27.048023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:29:27.048023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":641,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617707+0000","last_change":"2024-03-24T11:30:26.788751+0000","last_active":"2024-03-24T11:32:32.617707+0000","last_peered":"2024-03-24T11:32:32.617707+0000","last_clean":"2024-03-24T11:32:32.617707+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:32:32.617707+0000","last_undegraded":"2024-03-24T11:32:32.617707+0000","last_fullsized":"2024-03-24T11:32:32.617707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:30:26.788575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:30:26.788575+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":449,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619360+0000","last_change":"2024-03-24T11:31:15.842164+0000","last_active":"2024-03-24T11:32:32.619360+0000","last_peered":"2024-03-24T11:32:32.619360+0000","last_clean":"2024-03-24T11:32:32.619360+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:32:32.619360+0000","last_undegraded":"2024-03-24T11:32:32.619360+0000","last_fullsized":"2024-03-24T11:32:32.619360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.842054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.842054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":506,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617275+0000","last_change":"2024-03-24T11:31:10.289542+0000","last_active":"2024-03-24T11:32:32.617275+0000","last_peered":"2024-03-24T11:32:32.617275+0000","last_clean":"2024-03-24T11:32:32.617275+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:32:32.617275+0000","last_undegraded":"2024-03-24T11:32:32.617275+0000","last_fullsized":"2024-03-24T11:32:32.617275+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:10.289309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.289309+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"262'16688","reported_seq":16867,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.703813+0000","last_change":"2024-03-24T11:31:17.311636+0000","last_active":"2024-03-24T11:32:32.703813+0000","last_peered":"2024-03-24T11:32:32.703813+0000","last_clean":"2024-03-24T11:32:32.703813+0000","last_became_active":"2024-03-24T11:28:58.838515+0000","last_became_peered":"2024-03-24T11:28:58.838515+0000","last_unstale":"2024-03-24T11:32:32.703813+0000","last_undegraded":"2024-03-24T11:32:32.703813+0000","last_fullsized":"2024-03-24T11:32:32.703813+0000","mapping_epoch":211,"log_start":"248'12200","ondisk_log_start":"248'12200","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'11436","last_scrub_stamp":"2024-03-24T11:31:17.311316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:17.311316+0000","objects_scrubbed":5331,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:37.468415+0000","scrub_duration":2088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16688,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"213'39","reported_seq":538,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619046+0000","last_change":"2024-03-24T11:30:21.916752+0000","last_active":"2024-03-24T11:32:32.619046+0000","last_peered":"2024-03-24T11:32:32.619046+0000","last_clean":"2024-03-24T11:32:32.619046+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:32:32.619046+0000","last_undegraded":"2024-03-24T11:32:32.619046+0000","last_fullsized":"2024-03-24T11:32:32.619046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:30:21.916549+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":492,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704437+0000","last_change":"2024-03-24T11:32:23.906538+0000","last_active":"2024-03-24T11:32:32.704437+0000","last_peered":"2024-03-24T11:32:32.704437+0000","last_clean":"2024-03-24T11:32:32.704437+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:32:32.704437+0000","last_undegraded":"2024-03-24T11:32:32.704437+0000","last_fullsized":"2024-03-24T11:32:32.704437+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:32:23.906455+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.906455+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.906085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":469,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705622+0000","last_change":"2024-03-24T11:32:25.619271+0000","last_active":"2024-03-24T11:32:32.705622+0000","last_peered":"2024-03-24T11:32:32.705622+0000","last_clean":"2024-03-24T11:32:32.705622+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:32:32.705622+0000","last_undegraded":"2024-03-24T11:32:32.705622+0000","last_fullsized":"2024-03-24T11:32:32.705622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:25.619184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.619184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.019241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":465,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.673213+0000","last_change":"2024-03-24T11:31:35.151699+0000","last_active":"2024-03-24T11:32:32.673213+0000","last_peered":"2024-03-24T11:32:32.673213+0000","last_clean":"2024-03-24T11:32:32.673213+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:32:32.673213+0000","last_undegraded":"2024-03-24T11:32:32.673213+0000","last_fullsized":"2024-03-24T11:32:32.673213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:35.151566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:35.151566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:44.890575+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":603,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705864+0000","last_change":"2024-03-24T11:31:31.621753+0000","last_active":"2024-03-24T11:32:32.705864+0000","last_peered":"2024-03-24T11:32:32.705864+0000","last_clean":"2024-03-24T11:32:32.705864+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:32:32.705864+0000","last_undegraded":"2024-03-24T11:32:32.705864+0000","last_fullsized":"2024-03-24T11:32:32.705864+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.621646+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.866195+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"262'16988","reported_seq":17113,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617570+0000","last_change":"2024-03-24T11:30:35.379938+0000","last_active":"2024-03-24T11:32:32.617570+0000","last_peered":"2024-03-24T11:32:32.617570+0000","last_clean":"2024-03-24T11:32:32.617570+0000","last_became_active":"2024-03-24T11:30:35.355477+0000","last_became_peered":"2024-03-24T11:30:35.355477+0000","last_unstale":"2024-03-24T11:32:32.617570+0000","last_undegraded":"2024-03-24T11:32:32.617570+0000","last_fullsized":"2024-03-24T11:32:32.617570+0000","mapping_epoch":237,"log_start":"248'12851","ondisk_log_start":"248'12851","created":211,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:28:57.428807+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16988,"num_write_kb":543616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"213'19","reported_seq":496,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:37.784077+0000","last_change":"2024-03-24T11:32:37.784109+0000","last_active":"2024-03-24T11:32:37.784077+0000","last_peered":"2024-03-24T11:32:37.784077+0000","last_clean":"2024-03-24T11:32:37.784077+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:32:37.784077+0000","last_undegraded":"2024-03-24T11:32:37.784077+0000","last_fullsized":"2024-03-24T11:32:37.784077+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:37.784011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.784011+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:39.413955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":461,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617941+0000","last_change":"2024-03-24T11:31:29.730215+0000","last_active":"2024-03-24T11:32:32.617941+0000","last_peered":"2024-03-24T11:32:32.617941+0000","last_clean":"2024-03-24T11:32:32.617941+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:32:32.617941+0000","last_undegraded":"2024-03-24T11:32:32.617941+0000","last_fullsized":"2024-03-24T11:32:32.617941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.730078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.730078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:54.570128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707186+0000","last_change":"2024-03-24T11:30:10.252564+0000","last_active":"2024-03-24T11:32:32.707186+0000","last_peered":"2024-03-24T11:32:32.707186+0000","last_clean":"2024-03-24T11:32:32.707186+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:32:32.707186+0000","last_undegraded":"2024-03-24T11:32:32.707186+0000","last_fullsized":"2024-03-24T11:32:32.707186+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:10.252339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:30:10.252339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":433,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649117+0000","last_change":"2024-03-24T11:29:39.057894+0000","last_active":"2024-03-24T11:32:32.649117+0000","last_peered":"2024-03-24T11:32:32.649117+0000","last_clean":"2024-03-24T11:32:32.649117+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:32:32.649117+0000","last_undegraded":"2024-03-24T11:32:32.649117+0000","last_fullsized":"2024-03-24T11:32:32.649117+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:39.057644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:39.057644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":622,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704486+0000","last_change":"2024-03-24T11:32:24.651691+0000","last_active":"2024-03-24T11:32:32.704486+0000","last_peered":"2024-03-24T11:32:32.704486+0000","last_clean":"2024-03-24T11:32:32.704486+0000","last_became_active":"2024-03-24T11:28:43.190009+0000","last_became_peered":"2024-03-24T11:28:43.190009+0000","last_unstale":"2024-03-24T11:32:32.704486+0000","last_undegraded":"2024-03-24T11:32:32.704486+0000","last_fullsized":"2024-03-24T11:32:32.704486+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:24.651587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.651587+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.416321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"262'16788","reported_seq":16905,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617396+0000","last_change":"2024-03-24T11:28:58.457612+0000","last_active":"2024-03-24T11:32:32.617396+0000","last_peered":"2024-03-24T11:32:32.617396+0000","last_clean":"2024-03-24T11:32:32.617396+0000","last_became_active":"2024-03-24T11:28:58.457322+0000","last_became_peered":"2024-03-24T11:28:58.457322+0000","last_unstale":"2024-03-24T11:32:32.617396+0000","last_undegraded":"2024-03-24T11:32:32.617396+0000","last_fullsized":"2024-03-24T11:32:32.617396+0000","mapping_epoch":211,"log_start":"248'12600","ondisk_log_start":"248'12600","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:28:57.428807+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16788,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"213'34","reported_seq":523,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704378+0000","last_change":"2024-03-24T11:32:30.766638+0000","last_active":"2024-03-24T11:32:32.704378+0000","last_peered":"2024-03-24T11:32:32.704378+0000","last_clean":"2024-03-24T11:32:32.704378+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:32:32.704378+0000","last_undegraded":"2024-03-24T11:32:32.704378+0000","last_fullsized":"2024-03-24T11:32:32.704378+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:32:30.766549+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.766549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.554399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617395+0000","last_change":"2024-03-24T11:31:22.167292+0000","last_active":"2024-03-24T11:32:32.617395+0000","last_peered":"2024-03-24T11:32:32.617395+0000","last_clean":"2024-03-24T11:32:32.617395+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:32:32.617395+0000","last_undegraded":"2024-03-24T11:32:32.617395+0000","last_fullsized":"2024-03-24T11:32:32.617395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.167171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.167171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:40.906707+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":626,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619330+0000","last_change":"2024-03-24T11:32:18.783487+0000","last_active":"2024-03-24T11:32:32.619330+0000","last_peered":"2024-03-24T11:32:32.619330+0000","last_clean":"2024-03-24T11:32:32.619330+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:32:32.619330+0000","last_undegraded":"2024-03-24T11:32:32.619330+0000","last_fullsized":"2024-03-24T11:32:32.619330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:18.783339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.783339+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:29.190654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":449,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707021+0000","last_change":"2024-03-24T11:31:28.690070+0000","last_active":"2024-03-24T11:32:32.707021+0000","last_peered":"2024-03-24T11:32:32.707021+0000","last_clean":"2024-03-24T11:32:32.707021+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:32:32.707021+0000","last_undegraded":"2024-03-24T11:32:32.707021+0000","last_fullsized":"2024-03-24T11:32:32.707021+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.689946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.689946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:55.984173+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":594,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706347+0000","last_change":"2024-03-24T11:30:11.263503+0000","last_active":"2024-03-24T11:32:32.706347+0000","last_peered":"2024-03-24T11:32:32.706347+0000","last_clean":"2024-03-24T11:32:32.706347+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:32:32.706347+0000","last_undegraded":"2024-03-24T11:32:32.706347+0000","last_fullsized":"2024-03-24T11:32:32.706347+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:30:11.263251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.263251+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"262'16754","reported_seq":16960,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.808294+0000","last_change":"2024-03-24T11:32:32.808323+0000","last_active":"2024-03-24T11:32:32.808294+0000","last_peered":"2024-03-24T11:32:32.808294+0000","last_clean":"2024-03-24T11:32:32.808294+0000","last_became_active":"2024-03-24T11:30:35.369095+0000","last_became_peered":"2024-03-24T11:30:35.369095+0000","last_unstale":"2024-03-24T11:32:32.808294+0000","last_undegraded":"2024-03-24T11:32:32.808294+0000","last_fullsized":"2024-03-24T11:32:32.808294+0000","mapping_epoch":237,"log_start":"248'12300","ondisk_log_start":"248'12300","created":211,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'16754","last_scrub_stamp":"2024-03-24T11:32:32.808216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.808216+0000","objects_scrubbed":0,"log_size":4454,"log_dups_size":0,"ondisk_log_size":4454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:41.762945+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16754,"num_write_kb":536065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617834+0000","last_change":"2024-03-24T11:30:24.974463+0000","last_active":"2024-03-24T11:32:32.617834+0000","last_peered":"2024-03-24T11:32:32.617834+0000","last_clean":"2024-03-24T11:32:32.617834+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:32:32.617834+0000","last_undegraded":"2024-03-24T11:32:32.617834+0000","last_fullsized":"2024-03-24T11:32:32.617834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:24.974351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.974351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"213'30","reported_seq":491,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617446+0000","last_change":"2024-03-24T11:31:31.099444+0000","last_active":"2024-03-24T11:32:32.617446+0000","last_peered":"2024-03-24T11:32:32.617446+0000","last_clean":"2024-03-24T11:32:32.617446+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:32:32.617446+0000","last_undegraded":"2024-03-24T11:32:32.617446+0000","last_fullsized":"2024-03-24T11:32:32.617446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:31:31.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.099209+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.955611+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":445,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649241+0000","last_change":"2024-03-24T11:29:28.019634+0000","last_active":"2024-03-24T11:32:32.649241+0000","last_peered":"2024-03-24T11:32:32.649241+0000","last_clean":"2024-03-24T11:32:32.649241+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:32:32.649241+0000","last_undegraded":"2024-03-24T11:32:32.649241+0000","last_fullsized":"2024-03-24T11:32:32.649241+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:28.018667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:29:28.018667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":441,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618109+0000","last_change":"2024-03-24T11:30:17.941383+0000","last_active":"2024-03-24T11:32:32.618109+0000","last_peered":"2024-03-24T11:32:32.618109+0000","last_clean":"2024-03-24T11:32:32.618109+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:32:32.618109+0000","last_undegraded":"2024-03-24T11:32:32.618109+0000","last_fullsized":"2024-03-24T11:32:32.618109+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.941259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":607,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618640+0000","last_change":"2024-03-24T11:31:25.692917+0000","last_active":"2024-03-24T11:32:32.618640+0000","last_peered":"2024-03-24T11:32:32.618640+0000","last_clean":"2024-03-24T11:32:32.618640+0000","last_became_active":"2024-03-24T11:28:38.173086+0000","last_became_peered":"2024-03-24T11:28:38.173086+0000","last_unstale":"2024-03-24T11:32:32.618640+0000","last_undegraded":"2024-03-24T11:32:32.618640+0000","last_fullsized":"2024-03-24T11:32:32.618640+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.692790+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:28:38.972499+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.692790+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:41.349447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"262'16722","reported_seq":16839,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648966+0000","last_change":"2024-03-24T11:28:58.453980+0000","last_active":"2024-03-24T11:32:32.648966+0000","last_peered":"2024-03-24T11:32:32.648966+0000","last_clean":"2024-03-24T11:32:32.648966+0000","last_became_active":"2024-03-24T11:28:58.453692+0000","last_became_peered":"2024-03-24T11:28:58.453692+0000","last_unstale":"2024-03-24T11:32:32.648966+0000","last_undegraded":"2024-03-24T11:32:32.648966+0000","last_fullsized":"2024-03-24T11:32:32.648966+0000","mapping_epoch":211,"log_start":"248'12600","ondisk_log_start":"248'12600","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:28:57.428807+0000","objects_scrubbed":0,"log_size":4122,"log_dups_size":0,"ondisk_log_size":4122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16722,"num_write_kb":535104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":610,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617542+0000","last_change":"2024-03-24T11:31:08.290147+0000","last_active":"2024-03-24T11:32:32.617542+0000","last_peered":"2024-03-24T11:32:32.617542+0000","last_clean":"2024-03-24T11:32:32.617542+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:32:32.617542+0000","last_undegraded":"2024-03-24T11:32:32.617542+0000","last_fullsized":"2024-03-24T11:32:32.617542+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:08.289960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.289960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:36.006323+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"213'28","reported_seq":499,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:34.830405+0000","last_change":"2024-03-24T11:32:34.830458+0000","last_active":"2024-03-24T11:32:34.830405+0000","last_peered":"2024-03-24T11:32:34.830405+0000","last_clean":"2024-03-24T11:32:34.830405+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:32:34.830405+0000","last_undegraded":"2024-03-24T11:32:34.830405+0000","last_fullsized":"2024-03-24T11:32:34.830405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:32:34.830322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.830322+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.916433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":462,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704575+0000","last_change":"2024-03-24T11:30:21.493498+0000","last_active":"2024-03-24T11:32:32.704575+0000","last_peered":"2024-03-24T11:32:32.704575+0000","last_clean":"2024-03-24T11:32:32.704575+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:32:32.704575+0000","last_undegraded":"2024-03-24T11:32:32.704575+0000","last_fullsized":"2024-03-24T11:32:32.704575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:30:21.493357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:21.493357+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705391+0000","last_change":"2024-03-24T11:31:09.826116+0000","last_active":"2024-03-24T11:32:32.705391+0000","last_peered":"2024-03-24T11:32:32.705391+0000","last_clean":"2024-03-24T11:32:32.705391+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:32:32.705391+0000","last_undegraded":"2024-03-24T11:32:32.705391+0000","last_fullsized":"2024-03-24T11:32:32.705391+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:09.825994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:09.825994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":599,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649020+0000","last_change":"2024-03-24T11:31:18.144752+0000","last_active":"2024-03-24T11:32:32.649020+0000","last_peered":"2024-03-24T11:32:32.649020+0000","last_clean":"2024-03-24T11:32:32.649020+0000","last_became_active":"2024-03-24T11:28:38.174888+0000","last_became_peered":"2024-03-24T11:28:38.174888+0000","last_unstale":"2024-03-24T11:32:32.649020+0000","last_undegraded":"2024-03-24T11:32:32.649020+0000","last_fullsized":"2024-03-24T11:32:32.649020+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.144322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.144322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"262'16736","reported_seq":16964,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706838+0000","last_change":"2024-03-24T11:31:40.772483+0000","last_active":"2024-03-24T11:32:32.706838+0000","last_peered":"2024-03-24T11:32:32.706838+0000","last_clean":"2024-03-24T11:32:32.706838+0000","last_became_active":"2024-03-24T11:28:58.453179+0000","last_became_peered":"2024-03-24T11:28:58.453179+0000","last_unstale":"2024-03-24T11:32:32.706838+0000","last_undegraded":"2024-03-24T11:32:32.706838+0000","last_fullsized":"2024-03-24T11:32:32.706838+0000","mapping_epoch":211,"log_start":"248'12100","ondisk_log_start":"248'12100","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'13430","last_scrub_stamp":"2024-03-24T11:31:40.772373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:40.772373+0000","objects_scrubbed":3349,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.900093+0000","scrub_duration":1114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16736,"num_write_kb":535552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":627,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705424+0000","last_change":"2024-03-24T11:30:11.446821+0000","last_active":"2024-03-24T11:32:32.705424+0000","last_peered":"2024-03-24T11:32:32.705424+0000","last_clean":"2024-03-24T11:32:32.705424+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:32:32.705424+0000","last_undegraded":"2024-03-24T11:32:32.705424+0000","last_fullsized":"2024-03-24T11:32:32.705424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:30:11.446693+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:34.363907+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.446693+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"213'20","reported_seq":503,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704912+0000","last_change":"2024-03-24T11:32:21.588381+0000","last_active":"2024-03-24T11:32:32.704912+0000","last_peered":"2024-03-24T11:32:32.704912+0000","last_clean":"2024-03-24T11:32:32.704912+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:32:32.704912+0000","last_undegraded":"2024-03-24T11:32:32.704912+0000","last_fullsized":"2024-03-24T11:32:32.704912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:21.588239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.588239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.355301+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":472,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707091+0000","last_change":"2024-03-24T11:31:06.261980+0000","last_active":"2024-03-24T11:32:32.707091+0000","last_peered":"2024-03-24T11:32:32.707091+0000","last_clean":"2024-03-24T11:32:32.707091+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:32:32.707091+0000","last_undegraded":"2024-03-24T11:32:32.707091+0000","last_fullsized":"2024-03-24T11:32:32.707091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:06.261883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.261883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":450,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.703801+0000","last_change":"2024-03-24T11:32:22.994060+0000","last_active":"2024-03-24T11:32:32.703801+0000","last_peered":"2024-03-24T11:32:32.703801+0000","last_clean":"2024-03-24T11:32:32.703801+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:32:32.703801+0000","last_undegraded":"2024-03-24T11:32:32.703801+0000","last_fullsized":"2024-03-24T11:32:32.703801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:32:22.993851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:22.993851+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.104574+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":621,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705216+0000","last_change":"2024-03-24T11:31:42.062272+0000","last_active":"2024-03-24T11:32:32.705216+0000","last_peered":"2024-03-24T11:32:32.705216+0000","last_clean":"2024-03-24T11:32:32.705216+0000","last_became_active":"2024-03-24T11:28:38.178260+0000","last_became_peered":"2024-03-24T11:28:38.178260+0000","last_unstale":"2024-03-24T11:32:32.705216+0000","last_undegraded":"2024-03-24T11:32:32.705216+0000","last_fullsized":"2024-03-24T11:32:32.705216+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.062151+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:10.775672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"262'16664","reported_seq":16833,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619174+0000","last_change":"2024-03-24T11:31:32.691671+0000","last_active":"2024-03-24T11:32:32.619174+0000","last_peered":"2024-03-24T11:32:32.619174+0000","last_clean":"2024-03-24T11:32:32.619174+0000","last_became_active":"2024-03-24T11:30:35.355375+0000","last_became_peered":"2024-03-24T11:30:35.355375+0000","last_unstale":"2024-03-24T11:32:32.619174+0000","last_undegraded":"2024-03-24T11:32:32.619174+0000","last_fullsized":"2024-03-24T11:32:32.619174+0000","mapping_epoch":237,"log_start":"248'12300","ondisk_log_start":"248'12300","created":211,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'12743","last_scrub_stamp":"2024-03-24T11:31:32.691448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:32.691448+0000","objects_scrubbed":3950,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:56.928557+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16664,"num_write_kb":533248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":477,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704579+0000","last_change":"2024-03-24T11:31:33.075345+0000","last_active":"2024-03-24T11:32:32.704579+0000","last_peered":"2024-03-24T11:32:32.704579+0000","last_clean":"2024-03-24T11:32:32.704579+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:32:32.704579+0000","last_undegraded":"2024-03-24T11:32:32.704579+0000","last_fullsized":"2024-03-24T11:32:32.704579+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:33.075228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.075228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:56.584947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"213'29","reported_seq":508,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705158+0000","last_change":"2024-03-24T11:32:26.907881+0000","last_active":"2024-03-24T11:32:32.705158+0000","last_peered":"2024-03-24T11:32:32.705158+0000","last_clean":"2024-03-24T11:32:32.705158+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:32:32.705158+0000","last_undegraded":"2024-03-24T11:32:32.705158+0000","last_fullsized":"2024-03-24T11:32:32.705158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:32:26.907769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.907769+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:32.644339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":477,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705393+0000","last_change":"2024-03-24T11:31:25.294292+0000","last_active":"2024-03-24T11:32:32.705393+0000","last_peered":"2024-03-24T11:32:32.705393+0000","last_clean":"2024-03-24T11:32:32.705393+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:32:32.705393+0000","last_undegraded":"2024-03-24T11:32:32.705393+0000","last_fullsized":"2024-03-24T11:32:32.705393+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:25.294054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.294054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:44.339730+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":457,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704307+0000","last_change":"2024-03-24T11:31:20.405449+0000","last_active":"2024-03-24T11:32:32.704307+0000","last_peered":"2024-03-24T11:32:32.704307+0000","last_clean":"2024-03-24T11:32:32.704307+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:32:32.704307+0000","last_undegraded":"2024-03-24T11:32:32.704307+0000","last_fullsized":"2024-03-24T11:32:32.704307+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:20.405340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:20.405340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:42.309256+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":621,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704988+0000","last_change":"2024-03-24T11:31:18.894161+0000","last_active":"2024-03-24T11:32:32.704988+0000","last_peered":"2024-03-24T11:32:32.704988+0000","last_clean":"2024-03-24T11:32:32.704988+0000","last_became_active":"2024-03-24T11:28:38.174742+0000","last_became_peered":"2024-03-24T11:28:38.174742+0000","last_unstale":"2024-03-24T11:32:32.704988+0000","last_undegraded":"2024-03-24T11:32:32.704988+0000","last_fullsized":"2024-03-24T11:32:32.704988+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.894050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.894050+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:41.586923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"262'16698","reported_seq":16866,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704735+0000","last_change":"2024-03-24T11:31:29.059005+0000","last_active":"2024-03-24T11:32:32.704735+0000","last_peered":"2024-03-24T11:32:32.704735+0000","last_clean":"2024-03-24T11:32:32.704735+0000","last_became_active":"2024-03-24T11:28:58.838393+0000","last_became_peered":"2024-03-24T11:28:58.838393+0000","last_unstale":"2024-03-24T11:32:32.704735+0000","last_undegraded":"2024-03-24T11:32:32.704735+0000","last_fullsized":"2024-03-24T11:32:32.704735+0000","mapping_epoch":211,"log_start":"246'11300","ondisk_log_start":"246'11300","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'12402","last_scrub_stamp":"2024-03-24T11:31:29.058727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.058727+0000","objects_scrubbed":4353,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:38.888727+0000","scrub_duration":1454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16698,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707188+0000","last_change":"2024-03-24T11:30:27.391730+0000","last_active":"2024-03-24T11:32:32.707188+0000","last_peered":"2024-03-24T11:32:32.707188+0000","last_clean":"2024-03-24T11:32:32.707188+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:32:32.707188+0000","last_undegraded":"2024-03-24T11:32:32.707188+0000","last_fullsized":"2024-03-24T11:32:32.707188+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:30:27.391607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"213'24","reported_seq":493,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704466+0000","last_change":"2024-03-24T11:32:21.972798+0000","last_active":"2024-03-24T11:32:32.704466+0000","last_peered":"2024-03-24T11:32:32.704466+0000","last_clean":"2024-03-24T11:32:32.704466+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:32:32.704466+0000","last_undegraded":"2024-03-24T11:32:32.704466+0000","last_fullsized":"2024-03-24T11:32:32.704466+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:21.972644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.972644+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.486043+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":234,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.856934+0000","last_change":"2024-03-24T11:31:18.316986+0000","last_active":"2024-03-24T11:32:32.856934+0000","last_peered":"2024-03-24T11:32:32.856934+0000","last_clean":"2024-03-24T11:32:32.856934+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:32:32.856934+0000","last_undegraded":"2024-03-24T11:32:32.856934+0000","last_fullsized":"2024-03-24T11:32:32.856934+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:31:18.316859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.316859+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:39.547611+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":125,"num_read_kb":120,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":464,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704944+0000","last_change":"2024-03-24T11:31:21.629091+0000","last_active":"2024-03-24T11:32:32.704944+0000","last_peered":"2024-03-24T11:32:32.704944+0000","last_clean":"2024-03-24T11:32:32.704944+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:32:32.704944+0000","last_undegraded":"2024-03-24T11:32:32.704944+0000","last_fullsized":"2024-03-24T11:32:32.704944+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:21.628978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:21.628978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":465,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705208+0000","last_change":"2024-03-24T11:31:42.720752+0000","last_active":"2024-03-24T11:32:32.705208+0000","last_peered":"2024-03-24T11:32:32.705208+0000","last_clean":"2024-03-24T11:32:32.705208+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:32:32.705208+0000","last_undegraded":"2024-03-24T11:32:32.705208+0000","last_fullsized":"2024-03-24T11:32:32.705208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:42.720504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.720504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:00.102566+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":615,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704596+0000","last_change":"2024-03-24T11:32:26.629982+0000","last_active":"2024-03-24T11:32:32.704596+0000","last_peered":"2024-03-24T11:32:32.704596+0000","last_clean":"2024-03-24T11:32:32.704596+0000","last_became_active":"2024-03-24T11:28:38.173843+0000","last_became_peered":"2024-03-24T11:28:38.173843+0000","last_unstale":"2024-03-24T11:32:32.704596+0000","last_undegraded":"2024-03-24T11:32:32.704596+0000","last_fullsized":"2024-03-24T11:32:32.704596+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:26.629894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.629894+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.767066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"262'16372","reported_seq":16489,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617127+0000","last_change":"2024-03-24T11:28:58.452329+0000","last_active":"2024-03-24T11:32:32.617127+0000","last_peered":"2024-03-24T11:32:32.617127+0000","last_clean":"2024-03-24T11:32:32.617127+0000","last_became_active":"2024-03-24T11:28:58.452052+0000","last_became_peered":"2024-03-24T11:28:58.452052+0000","last_unstale":"2024-03-24T11:32:32.617127+0000","last_undegraded":"2024-03-24T11:32:32.617127+0000","last_fullsized":"2024-03-24T11:32:32.617127+0000","mapping_epoch":211,"log_start":"248'12200","ondisk_log_start":"248'12200","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:28:57.428807+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16372,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.673234+0000","last_change":"2024-03-24T11:31:04.854602+0000","last_active":"2024-03-24T11:32:32.673234+0000","last_peered":"2024-03-24T11:32:32.673234+0000","last_clean":"2024-03-24T11:32:32.673234+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:32:32.673234+0000","last_undegraded":"2024-03-24T11:32:32.673234+0000","last_fullsized":"2024-03-24T11:32:32.673234+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:04.854485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:31:04.854485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"213'36","reported_seq":693,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705439+0000","last_change":"2024-03-24T11:32:25.853007+0000","last_active":"2024-03-24T11:32:32.705439+0000","last_peered":"2024-03-24T11:32:32.705439+0000","last_clean":"2024-03-24T11:32:32.705439+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:32:32.705439+0000","last_undegraded":"2024-03-24T11:32:32.705439+0000","last_fullsized":"2024-03-24T11:32:32.705439+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'36","last_scrub_stamp":"2024-03-24T11:32:25.852913+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.852913+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.114595+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":459,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648611+0000","last_change":"2024-03-24T11:32:28.089047+0000","last_active":"2024-03-24T11:32:32.648611+0000","last_peered":"2024-03-24T11:32:32.648611+0000","last_clean":"2024-03-24T11:32:32.648611+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:32:32.648611+0000","last_undegraded":"2024-03-24T11:32:32.648611+0000","last_fullsized":"2024-03-24T11:32:32.648611+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.088969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.088969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.266746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":368,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.703958+0000","last_change":"2024-03-24T11:31:07.885631+0000","last_active":"2024-03-24T11:32:32.703958+0000","last_peered":"2024-03-24T11:32:32.703958+0000","last_clean":"2024-03-24T11:32:32.703958+0000","last_became_active":"2024-03-24T11:30:35.355009+0000","last_became_peered":"2024-03-24T11:30:35.355009+0000","last_unstale":"2024-03-24T11:32:32.703958+0000","last_undegraded":"2024-03-24T11:32:32.703958+0000","last_fullsized":"2024-03-24T11:32:32.703958+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:07.885508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:07.885508+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":465,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707020+0000","last_change":"2024-03-24T11:31:26.676287+0000","last_active":"2024-03-24T11:32:32.707020+0000","last_peered":"2024-03-24T11:32:32.707020+0000","last_clean":"2024-03-24T11:32:32.707020+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:32:32.707020+0000","last_undegraded":"2024-03-24T11:32:32.707020+0000","last_fullsized":"2024-03-24T11:32:32.707020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:26.676209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:31:26.676209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:36.838458+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"262'16596","reported_seq":16778,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706121+0000","last_change":"2024-03-24T11:31:12.220650+0000","last_active":"2024-03-24T11:32:32.706121+0000","last_peered":"2024-03-24T11:32:32.706121+0000","last_clean":"2024-03-24T11:32:32.706121+0000","last_became_active":"2024-03-24T11:28:58.451843+0000","last_became_peered":"2024-03-24T11:28:58.451843+0000","last_unstale":"2024-03-24T11:32:32.706121+0000","last_undegraded":"2024-03-24T11:32:32.706121+0000","last_fullsized":"2024-03-24T11:32:32.706121+0000","mapping_epoch":211,"log_start":"248'12000","ondisk_log_start":"248'12000","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'11010","last_scrub_stamp":"2024-03-24T11:31:12.219664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:12.219664+0000","objects_scrubbed":5647,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16596,"num_write_kb":531072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":469,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618068+0000","last_change":"2024-03-24T11:31:31.994930+0000","last_active":"2024-03-24T11:32:32.618068+0000","last_peered":"2024-03-24T11:32:32.618068+0000","last_clean":"2024-03-24T11:32:32.618068+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:32:32.618068+0000","last_undegraded":"2024-03-24T11:32:32.618068+0000","last_fullsized":"2024-03-24T11:32:32.618068+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:31.994604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.994604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:42.774652+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"213'17","reported_seq":473,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618026+0000","last_change":"2024-03-24T11:31:05.833649+0000","last_active":"2024-03-24T11:32:32.618026+0000","last_peered":"2024-03-24T11:32:32.618026+0000","last_clean":"2024-03-24T11:32:32.618026+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:32:32.618026+0000","last_undegraded":"2024-03-24T11:32:32.618026+0000","last_fullsized":"2024-03-24T11:32:32.618026+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'17","last_scrub_stamp":"2024-03-24T11:31:05.833501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:05.833501+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":486,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617775+0000","last_change":"2024-03-24T11:32:18.855130+0000","last_active":"2024-03-24T11:32:32.617775+0000","last_peered":"2024-03-24T11:32:32.617775+0000","last_clean":"2024-03-24T11:32:32.617775+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:32:32.617775+0000","last_undegraded":"2024-03-24T11:32:32.617775+0000","last_fullsized":"2024-03-24T11:32:32.617775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:18.854979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.854979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:20.771742+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":477,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704088+0000","last_change":"2024-03-24T11:31:29.443192+0000","last_active":"2024-03-24T11:32:32.704088+0000","last_peered":"2024-03-24T11:32:32.704088+0000","last_clean":"2024-03-24T11:32:32.704088+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:32:32.704088+0000","last_undegraded":"2024-03-24T11:32:32.704088+0000","last_fullsized":"2024-03-24T11:32:32.704088+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.443013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.443013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.737539+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705568+0000","last_change":"2024-03-24T11:31:13.874185+0000","last_active":"2024-03-24T11:32:32.705568+0000","last_peered":"2024-03-24T11:32:32.705568+0000","last_clean":"2024-03-24T11:32:32.705568+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:32:32.705568+0000","last_undegraded":"2024-03-24T11:32:32.705568+0000","last_fullsized":"2024-03-24T11:32:32.705568+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:13.874068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.874068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:35.769261+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":368,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704946+0000","last_change":"2024-03-24T11:31:08.903891+0000","last_active":"2024-03-24T11:32:32.704946+0000","last_peered":"2024-03-24T11:32:32.704946+0000","last_clean":"2024-03-24T11:32:32.704946+0000","last_became_active":"2024-03-24T11:30:35.356166+0000","last_became_peered":"2024-03-24T11:30:35.356166+0000","last_unstale":"2024-03-24T11:32:32.704946+0000","last_undegraded":"2024-03-24T11:32:32.704946+0000","last_fullsized":"2024-03-24T11:32:32.704946+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:08.903763+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.903763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"262'19384","reported_seq":19617,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704782+0000","last_change":"2024-03-24T11:31:59.155853+0000","last_active":"2024-03-24T11:32:32.704782+0000","last_peered":"2024-03-24T11:32:32.704782+0000","last_clean":"2024-03-24T11:32:32.704782+0000","last_became_active":"2024-03-24T11:28:58.451809+0000","last_became_peered":"2024-03-24T11:28:58.451809+0000","last_unstale":"2024-03-24T11:32:32.704782+0000","last_undegraded":"2024-03-24T11:32:32.704782+0000","last_fullsized":"2024-03-24T11:32:32.704782+0000","mapping_epoch":211,"log_start":"250'14003","ondisk_log_start":"250'14003","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'17308","last_scrub_stamp":"2024-03-24T11:31:59.155678+0000","last_deep_scrub":"252'17308","last_deep_scrub_stamp":"2024-03-24T11:31:59.155678+0000","last_clean_scrub_stamp":"2024-03-24T11:31:59.155678+0000","objects_scrubbed":2628,"log_size":5381,"log_dups_size":0,"ondisk_log_size":5381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":18,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:14.257219+0000","scrub_duration":17586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16756,"num_write_kb":536192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704327+0000","last_change":"2024-03-24T11:31:28.613941+0000","last_active":"2024-03-24T11:32:32.704327+0000","last_peered":"2024-03-24T11:32:32.704327+0000","last_clean":"2024-03-24T11:32:32.704327+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:32:32.704327+0000","last_undegraded":"2024-03-24T11:32:32.704327+0000","last_fullsized":"2024-03-24T11:32:32.704327+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.613752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.613752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:40.970305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"213'40","reported_seq":696,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705588+0000","last_change":"2024-03-24T11:31:30.466347+0000","last_active":"2024-03-24T11:32:32.705588+0000","last_peered":"2024-03-24T11:32:32.705588+0000","last_clean":"2024-03-24T11:32:32.705588+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:32:32.705588+0000","last_undegraded":"2024-03-24T11:32:32.705588+0000","last_fullsized":"2024-03-24T11:32:32.705588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'40","last_scrub_stamp":"2024-03-24T11:31:30.466176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.466176+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:42.245165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":461,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618125+0000","last_change":"2024-03-24T11:31:05.775644+0000","last_active":"2024-03-24T11:32:32.618125+0000","last_peered":"2024-03-24T11:32:32.618125+0000","last_clean":"2024-03-24T11:32:32.618125+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:32:32.618125+0000","last_undegraded":"2024-03-24T11:32:32.618125+0000","last_fullsized":"2024-03-24T11:32:32.618125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:05.775534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:05.775534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":480,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706584+0000","last_change":"2024-03-24T11:32:28.707278+0000","last_active":"2024-03-24T11:32:32.706584+0000","last_peered":"2024-03-24T11:32:32.706584+0000","last_clean":"2024-03-24T11:32:32.706584+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:32:32.706584+0000","last_undegraded":"2024-03-24T11:32:32.706584+0000","last_fullsized":"2024-03-24T11:32:32.706584+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:32:28.707184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.707184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:33.691220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":433,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648873+0000","last_change":"2024-03-24T11:29:18.915411+0000","last_active":"2024-03-24T11:32:32.648873+0000","last_peered":"2024-03-24T11:32:32.648873+0000","last_clean":"2024-03-24T11:32:32.648873+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:32:32.648873+0000","last_undegraded":"2024-03-24T11:32:32.648873+0000","last_fullsized":"2024-03-24T11:32:32.648873+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:18.915289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:18.915289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":615,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706343+0000","last_change":"2024-03-24T11:32:29.691475+0000","last_active":"2024-03-24T11:32:32.706343+0000","last_peered":"2024-03-24T11:32:32.706343+0000","last_clean":"2024-03-24T11:32:32.706343+0000","last_became_active":"2024-03-24T11:28:38.173832+0000","last_became_peered":"2024-03-24T11:28:38.173832+0000","last_unstale":"2024-03-24T11:32:32.706343+0000","last_undegraded":"2024-03-24T11:32:32.706343+0000","last_fullsized":"2024-03-24T11:32:32.706343+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:29.691383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:29.691383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.888681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"262'16780","reported_seq":16979,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705256+0000","last_change":"2024-03-24T11:30:24.852497+0000","last_active":"2024-03-24T11:32:32.705256+0000","last_peered":"2024-03-24T11:32:32.705256+0000","last_clean":"2024-03-24T11:32:32.705256+0000","last_became_active":"2024-03-24T11:28:58.455095+0000","last_became_peered":"2024-03-24T11:28:58.455095+0000","last_unstale":"2024-03-24T11:32:32.705256+0000","last_undegraded":"2024-03-24T11:32:32.705256+0000","last_fullsized":"2024-03-24T11:32:32.705256+0000","mapping_epoch":211,"log_start":"248'12200","ondisk_log_start":"248'12200","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'7370","last_scrub_stamp":"2024-03-24T11:30:24.851991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.851991+0000","objects_scrubbed":7249,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16780,"num_write_kb":536960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"213'15","reported_seq":482,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705340+0000","last_change":"2024-03-24T11:30:25.435813+0000","last_active":"2024-03-24T11:32:32.705340+0000","last_peered":"2024-03-24T11:32:32.705340+0000","last_clean":"2024-03-24T11:32:32.705340+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:32:32.705340+0000","last_undegraded":"2024-03-24T11:32:32.705340+0000","last_fullsized":"2024-03-24T11:32:32.705340+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:25.435671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:25.435671+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":475,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704984+0000","last_change":"2024-03-24T11:31:20.465180+0000","last_active":"2024-03-24T11:32:32.704984+0000","last_peered":"2024-03-24T11:32:32.704984+0000","last_clean":"2024-03-24T11:32:32.704984+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:32:32.704984+0000","last_undegraded":"2024-03-24T11:32:32.704984+0000","last_fullsized":"2024-03-24T11:32:32.704984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:20.465096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:20.465096+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.122475+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":469,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617630+0000","last_change":"2024-03-24T11:32:27.943222+0000","last_active":"2024-03-24T11:32:32.617630+0000","last_peered":"2024-03-24T11:32:32.617630+0000","last_clean":"2024-03-24T11:32:32.617630+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:32:32.617630+0000","last_undegraded":"2024-03-24T11:32:32.617630+0000","last_fullsized":"2024-03-24T11:32:32.617630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.943114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.943114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.987411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":449,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705226+0000","last_change":"2024-03-24T11:31:23.762948+0000","last_active":"2024-03-24T11:32:32.705226+0000","last_peered":"2024-03-24T11:32:32.705226+0000","last_clean":"2024-03-24T11:32:32.705226+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:32:32.705226+0000","last_undegraded":"2024-03-24T11:32:32.705226+0000","last_fullsized":"2024-03-24T11:32:32.705226+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:23.762486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:23.762486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:43.614566+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":446,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618893+0000","last_change":"2024-03-24T11:31:33.687541+0000","last_active":"2024-03-24T11:32:32.618893+0000","last_peered":"2024-03-24T11:32:32.618893+0000","last_clean":"2024-03-24T11:32:32.618893+0000","last_became_active":"2024-03-24T11:28:43.192246+0000","last_became_peered":"2024-03-24T11:28:43.192246+0000","last_unstale":"2024-03-24T11:32:32.618893+0000","last_undegraded":"2024-03-24T11:32:32.618893+0000","last_fullsized":"2024-03-24T11:32:32.618893+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:33.687261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.687261+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:43.601587+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"262'16532","reported_seq":16713,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706849+0000","last_change":"2024-03-24T11:31:13.145889+0000","last_active":"2024-03-24T11:32:32.706849+0000","last_peered":"2024-03-24T11:32:32.706849+0000","last_clean":"2024-03-24T11:32:32.706849+0000","last_became_active":"2024-03-24T11:28:58.453004+0000","last_became_peered":"2024-03-24T11:28:58.453004+0000","last_unstale":"2024-03-24T11:32:32.706849+0000","last_undegraded":"2024-03-24T11:32:32.706849+0000","last_fullsized":"2024-03-24T11:32:32.706849+0000","mapping_epoch":211,"log_start":"248'11900","ondisk_log_start":"248'11900","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'11030","last_scrub_stamp":"2024-03-24T11:31:13.145731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.145731+0000","objects_scrubbed":5569,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:39.730763+0000","scrub_duration":1745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16532,"num_write_kb":529024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"213'44","reported_seq":537,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705765+0000","last_change":"2024-03-24T11:31:21.903188+0000","last_active":"2024-03-24T11:32:32.705765+0000","last_peered":"2024-03-24T11:32:32.705765+0000","last_clean":"2024-03-24T11:32:32.705765+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:32:32.705765+0000","last_undegraded":"2024-03-24T11:32:32.705765+0000","last_fullsized":"2024-03-24T11:32:32.705765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:31:21.903023+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:31:21.903023+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:37.797080+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":64,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707105+0000","last_change":"2024-03-24T11:30:35.358614+0000","last_active":"2024-03-24T11:32:32.707105+0000","last_peered":"2024-03-24T11:32:32.707105+0000","last_clean":"2024-03-24T11:32:32.707105+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:32:32.707105+0000","last_undegraded":"2024-03-24T11:32:32.707105+0000","last_fullsized":"2024-03-24T11:32:32.707105+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:15.633276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:15.633276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":469,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617577+0000","last_change":"2024-03-24T11:31:22.849153+0000","last_active":"2024-03-24T11:32:32.617577+0000","last_peered":"2024-03-24T11:32:32.617577+0000","last_clean":"2024-03-24T11:32:32.617577+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:32:32.617577+0000","last_undegraded":"2024-03-24T11:32:32.617577+0000","last_fullsized":"2024-03-24T11:32:32.617577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.849049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.849049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:48.408647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618109+0000","last_change":"2024-03-24T11:31:32.996560+0000","last_active":"2024-03-24T11:32:32.618109+0000","last_peered":"2024-03-24T11:32:32.618109+0000","last_clean":"2024-03-24T11:32:32.618109+0000","last_became_active":"2024-03-24T11:30:35.355210+0000","last_became_peered":"2024-03-24T11:30:35.355210+0000","last_unstale":"2024-03-24T11:32:32.618109+0000","last_undegraded":"2024-03-24T11:32:32.618109+0000","last_fullsized":"2024-03-24T11:32:32.618109+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:32.996416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:32.996416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:41.925710+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":620,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704972+0000","last_change":"2024-03-24T11:31:12.838142+0000","last_active":"2024-03-24T11:32:32.704972+0000","last_peered":"2024-03-24T11:32:32.704972+0000","last_clean":"2024-03-24T11:32:32.704972+0000","last_became_active":"2024-03-24T11:28:43.191874+0000","last_became_peered":"2024-03-24T11:28:43.191874+0000","last_unstale":"2024-03-24T11:32:32.704972+0000","last_undegraded":"2024-03-24T11:32:32.704972+0000","last_fullsized":"2024-03-24T11:32:32.704972+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:12.838005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:12.838005+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"262'16484","reported_seq":16700,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705100+0000","last_change":"2024-03-24T11:31:16.352897+0000","last_active":"2024-03-24T11:32:32.705100+0000","last_peered":"2024-03-24T11:32:32.705100+0000","last_clean":"2024-03-24T11:32:32.705100+0000","last_became_active":"2024-03-24T11:28:58.455900+0000","last_became_peered":"2024-03-24T11:28:58.455900+0000","last_unstale":"2024-03-24T11:32:32.705100+0000","last_undegraded":"2024-03-24T11:32:32.705100+0000","last_fullsized":"2024-03-24T11:32:32.705100+0000","mapping_epoch":211,"log_start":"248'12000","ondisk_log_start":"248'12000","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'11195","last_scrub_stamp":"2024-03-24T11:31:16.352387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:16.352387+0000","objects_scrubbed":5349,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16484,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":469,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617471+0000","last_change":"2024-03-24T11:31:36.047503+0000","last_active":"2024-03-24T11:32:32.617471+0000","last_peered":"2024-03-24T11:32:32.617471+0000","last_clean":"2024-03-24T11:32:32.617471+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:32:32.617471+0000","last_undegraded":"2024-03-24T11:32:32.617471+0000","last_fullsized":"2024-03-24T11:32:32.617471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:36.047102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.047102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:37.614251+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"213'18","reported_seq":489,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618238+0000","last_change":"2024-03-24T11:31:11.798229+0000","last_active":"2024-03-24T11:32:32.618238+0000","last_peered":"2024-03-24T11:32:32.618238+0000","last_clean":"2024-03-24T11:32:32.618238+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:32:32.618238+0000","last_undegraded":"2024-03-24T11:32:32.618238+0000","last_fullsized":"2024-03-24T11:32:32.618238+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.798089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.798089+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:35.210465+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":64,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619101+0000","last_change":"2024-03-24T11:30:35.363477+0000","last_active":"2024-03-24T11:32:32.619101+0000","last_peered":"2024-03-24T11:32:32.619101+0000","last_clean":"2024-03-24T11:32:32.619101+0000","last_became_active":"2024-03-24T11:30:35.356055+0000","last_became_peered":"2024-03-24T11:30:35.356055+0000","last_unstale":"2024-03-24T11:32:32.619101+0000","last_undegraded":"2024-03-24T11:32:32.619101+0000","last_fullsized":"2024-03-24T11:32:32.619101+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:07.709842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:30:07.709842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":433,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648615+0000","last_change":"2024-03-24T11:30:12.206577+0000","last_active":"2024-03-24T11:32:32.648615+0000","last_peered":"2024-03-24T11:32:32.648615+0000","last_clean":"2024-03-24T11:32:32.648615+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:32:32.648615+0000","last_undegraded":"2024-03-24T11:32:32.648615+0000","last_fullsized":"2024-03-24T11:32:32.648615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:12.206287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:12.206287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":604,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648912+0000","last_change":"2024-03-24T11:29:10.818100+0000","last_active":"2024-03-24T11:32:32.648912+0000","last_peered":"2024-03-24T11:32:32.648912+0000","last_clean":"2024-03-24T11:32:32.648912+0000","last_became_active":"2024-03-24T11:28:43.189494+0000","last_became_peered":"2024-03-24T11:28:43.189494+0000","last_unstale":"2024-03-24T11:32:32.648912+0000","last_undegraded":"2024-03-24T11:32:32.648912+0000","last_fullsized":"2024-03-24T11:32:32.648912+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:29:10.817950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:29:10.817950+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"262'16860","reported_seq":17021,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704220+0000","last_change":"2024-03-24T11:31:36.787070+0000","last_active":"2024-03-24T11:32:32.704220+0000","last_peered":"2024-03-24T11:32:32.704220+0000","last_clean":"2024-03-24T11:32:32.704220+0000","last_became_active":"2024-03-24T11:28:58.453282+0000","last_became_peered":"2024-03-24T11:28:58.453282+0000","last_unstale":"2024-03-24T11:32:32.704220+0000","last_undegraded":"2024-03-24T11:32:32.704220+0000","last_fullsized":"2024-03-24T11:32:32.704220+0000","mapping_epoch":211,"log_start":"246'11500","ondisk_log_start":"246'11500","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'13278","last_scrub_stamp":"2024-03-24T11:31:36.786723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:57.428807+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.786723+0000","objects_scrubbed":3633,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:55.188778+0000","scrub_duration":1205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16860,"num_write_kb":539520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"238'9","reported_seq":121,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706179+0000","last_change":"2024-03-24T11:31:30.056940+0000","last_active":"2024-03-24T11:32:32.706179+0000","last_peered":"2024-03-24T11:32:32.706179+0000","last_clean":"2024-03-24T11:32:32.706179+0000","last_became_active":"2024-03-24T11:30:35.359635+0000","last_became_peered":"2024-03-24T11:30:35.359635+0000","last_unstale":"2024-03-24T11:32:32.706179+0000","last_undegraded":"2024-03-24T11:32:32.706179+0000","last_fullsized":"2024-03-24T11:32:32.706179+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'9","last_scrub_stamp":"2024-03-24T11:31:30.056809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.056809+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:37.637183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"213'34","reported_seq":515,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705610+0000","last_change":"2024-03-24T11:31:12.435804+0000","last_active":"2024-03-24T11:32:32.705610+0000","last_peered":"2024-03-24T11:32:32.705610+0000","last_clean":"2024-03-24T11:32:32.705610+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:32:32.705610+0000","last_undegraded":"2024-03-24T11:32:32.705610+0000","last_fullsized":"2024-03-24T11:32:32.705610+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:31:12.435639+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:31:12.435639+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:39.034351+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":266,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619096+0000","last_change":"2024-03-24T11:30:35.356491+0000","last_active":"2024-03-24T11:32:32.619096+0000","last_peered":"2024-03-24T11:32:32.619096+0000","last_clean":"2024-03-24T11:32:32.619096+0000","last_became_active":"2024-03-24T11:30:35.356100+0000","last_became_peered":"2024-03-24T11:30:35.356100+0000","last_unstale":"2024-03-24T11:32:32.619096+0000","last_undegraded":"2024-03-24T11:32:32.619096+0000","last_fullsized":"2024-03-24T11:32:32.619096+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:30:11.625473+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.625473+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":457,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706837+0000","last_change":"2024-03-24T11:31:14.458586+0000","last_active":"2024-03-24T11:32:32.706837+0000","last_peered":"2024-03-24T11:32:32.706837+0000","last_clean":"2024-03-24T11:32:32.706837+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:32:32.706837+0000","last_undegraded":"2024-03-24T11:32:32.706837+0000","last_fullsized":"2024-03-24T11:32:32.706837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.458470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.458470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:37.499878+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":531,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618630+0000","last_change":"2024-03-24T11:31:04.735517+0000","last_active":"2024-03-24T11:32:32.618630+0000","last_peered":"2024-03-24T11:32:32.618630+0000","last_clean":"2024-03-24T11:32:32.618630+0000","last_became_active":"2024-03-24T11:30:35.354637+0000","last_became_peered":"2024-03-24T11:30:35.354637+0000","last_unstale":"2024-03-24T11:32:32.618630+0000","last_undegraded":"2024-03-24T11:32:32.618630+0000","last_fullsized":"2024-03-24T11:32:32.618630+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:04.735433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:04.735433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:34.114928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"213'15","reported_seq":461,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649427+0000","last_change":"2024-03-24T11:30:14.150582+0000","last_active":"2024-03-24T11:32:32.649427+0000","last_peered":"2024-03-24T11:32:32.649427+0000","last_clean":"2024-03-24T11:32:32.649427+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:32:32.649427+0000","last_undegraded":"2024-03-24T11:32:32.649427+0000","last_fullsized":"2024-03-24T11:32:32.649427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:14.150454+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.150454+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":476,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704375+0000","last_change":"2024-03-24T11:31:16.803402+0000","last_active":"2024-03-24T11:32:32.704375+0000","last_peered":"2024-03-24T11:32:32.704375+0000","last_clean":"2024-03-24T11:32:32.704375+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:32:32.704375+0000","last_undegraded":"2024-03-24T11:32:32.704375+0000","last_fullsized":"2024-03-24T11:32:32.704375+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:16.802841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:16.802841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":452,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705569+0000","last_change":"2024-03-24T11:30:13.245185+0000","last_active":"2024-03-24T11:32:32.705569+0000","last_peered":"2024-03-24T11:32:32.705569+0000","last_clean":"2024-03-24T11:32:32.705569+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:32:32.705569+0000","last_undegraded":"2024-03-24T11:32:32.705569+0000","last_fullsized":"2024-03-24T11:32:32.705569+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:13.245067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:30:13.245067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":80,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.920055+0000","last_change":"2024-03-24T11:32:32.920081+0000","last_active":"2024-03-24T11:32:32.920055+0000","last_peered":"2024-03-24T11:32:32.920055+0000","last_clean":"2024-03-24T11:32:32.920055+0000","last_became_active":"2024-03-24T11:30:35.353212+0000","last_became_peered":"2024-03-24T11:30:35.353212+0000","last_unstale":"2024-03-24T11:32:32.920055+0000","last_undegraded":"2024-03-24T11:32:32.920055+0000","last_fullsized":"2024-03-24T11:32:32.920055+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.919963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:34.622437+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":446,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706233+0000","last_change":"2024-03-24T11:31:24.533967+0000","last_active":"2024-03-24T11:32:32.706233+0000","last_peered":"2024-03-24T11:32:32.706233+0000","last_clean":"2024-03-24T11:32:32.706233+0000","last_became_active":"2024-03-24T11:28:43.189410+0000","last_became_peered":"2024-03-24T11:28:43.189410+0000","last_unstale":"2024-03-24T11:32:32.706233+0000","last_undegraded":"2024-03-24T11:32:32.706233+0000","last_fullsized":"2024-03-24T11:32:32.706233+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_clean_scrub_stamp":"2024-03-24T11:31:24.533874+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:51.862202+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":622,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:33.790843+0000","last_change":"2024-03-24T11:32:33.790867+0000","last_active":"2024-03-24T11:32:33.790843+0000","last_peered":"2024-03-24T11:32:33.790843+0000","last_clean":"2024-03-24T11:32:33.790843+0000","last_became_active":"2024-03-24T11:28:43.192449+0000","last_became_peered":"2024-03-24T11:28:43.192449+0000","last_unstale":"2024-03-24T11:32:33.790843+0000","last_undegraded":"2024-03-24T11:32:33.790843+0000","last_fullsized":"2024-03-24T11:32:33.790843+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.790777+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.025287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"213'19","reported_seq":493,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705154+0000","last_change":"2024-03-24T11:32:32.023601+0000","last_active":"2024-03-24T11:32:32.705154+0000","last_peered":"2024-03-24T11:32:32.705154+0000","last_clean":"2024-03-24T11:32:32.705154+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:32:32.705154+0000","last_undegraded":"2024-03-24T11:32:32.705154+0000","last_fullsized":"2024-03-24T11:32:32.705154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:32.023502+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.023502+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.100674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":64,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618118+0000","last_change":"2024-03-24T11:30:35.359322+0000","last_active":"2024-03-24T11:32:32.618118+0000","last_peered":"2024-03-24T11:32:32.618118+0000","last_clean":"2024-03-24T11:32:32.618118+0000","last_became_active":"2024-03-24T11:30:35.358928+0000","last_became_peered":"2024-03-24T11:30:35.358928+0000","last_unstale":"2024-03-24T11:32:32.618118+0000","last_undegraded":"2024-03-24T11:32:32.618118+0000","last_fullsized":"2024-03-24T11:32:32.618118+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:21.749544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:21.749544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":436,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649400+0000","last_change":"2024-03-24T11:29:26.014447+0000","last_active":"2024-03-24T11:32:32.649400+0000","last_peered":"2024-03-24T11:32:32.649400+0000","last_clean":"2024-03-24T11:32:32.649400+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:32:32.649400+0000","last_undegraded":"2024-03-24T11:32:32.649400+0000","last_fullsized":"2024-03-24T11:32:32.649400+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:26.014199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:29:26.014199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":464,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.673075+0000","last_change":"2024-03-24T11:32:26.898659+0000","last_active":"2024-03-24T11:32:32.673075+0000","last_peered":"2024-03-24T11:32:32.673075+0000","last_clean":"2024-03-24T11:32:32.673075+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:32:32.673075+0000","last_undegraded":"2024-03-24T11:32:32.673075+0000","last_fullsized":"2024-03-24T11:32:32.673075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:26.898583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.898583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.612791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"213'15","reported_seq":489,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707187+0000","last_change":"2024-03-24T11:31:27.541622+0000","last_active":"2024-03-24T11:32:32.707187+0000","last_peered":"2024-03-24T11:32:32.707187+0000","last_clean":"2024-03-24T11:32:32.707187+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:32:32.707187+0000","last_undegraded":"2024-03-24T11:32:32.707187+0000","last_fullsized":"2024-03-24T11:32:32.707187+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:27.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.541510+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:52.331987+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704843+0000","last_change":"2024-03-24T11:30:26.479018+0000","last_active":"2024-03-24T11:32:32.704843+0000","last_peered":"2024-03-24T11:32:32.704843+0000","last_clean":"2024-03-24T11:32:32.704843+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:32:32.704843+0000","last_undegraded":"2024-03-24T11:32:32.704843+0000","last_fullsized":"2024-03-24T11:32:32.704843+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:26.478900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:30:26.478900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":617,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649539+0000","last_change":"2024-03-24T11:31:13.201069+0000","last_active":"2024-03-24T11:32:32.649539+0000","last_peered":"2024-03-24T11:32:32.649539+0000","last_clean":"2024-03-24T11:32:32.649539+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:32:32.649539+0000","last_undegraded":"2024-03-24T11:32:32.649539+0000","last_fullsized":"2024-03-24T11:32:32.649539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:13.200503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.200503+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":448,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619216+0000","last_change":"2024-03-24T11:32:27.764304+0000","last_active":"2024-03-24T11:32:32.619216+0000","last_peered":"2024-03-24T11:32:32.619216+0000","last_clean":"2024-03-24T11:32:32.619216+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:32:32.619216+0000","last_undegraded":"2024-03-24T11:32:32.619216+0000","last_fullsized":"2024-03-24T11:32:32.619216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.764203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.764203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.837847+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":446,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705978+0000","last_change":"2024-03-24T11:31:25.526652+0000","last_active":"2024-03-24T11:32:32.705978+0000","last_peered":"2024-03-24T11:32:32.705978+0000","last_clean":"2024-03-24T11:32:32.705978+0000","last_became_active":"2024-03-24T11:28:43.188941+0000","last_became_peered":"2024-03-24T11:28:43.188941+0000","last_unstale":"2024-03-24T11:32:32.705978+0000","last_undegraded":"2024-03-24T11:32:32.705978+0000","last_fullsized":"2024-03-24T11:32:32.705978+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.526522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"213'20","reported_seq":470,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649395+0000","last_change":"2024-03-24T11:29:32.055362+0000","last_active":"2024-03-24T11:32:32.649395+0000","last_peered":"2024-03-24T11:32:32.649395+0000","last_clean":"2024-03-24T11:32:32.649395+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:32:32.649395+0000","last_undegraded":"2024-03-24T11:32:32.649395+0000","last_fullsized":"2024-03-24T11:32:32.649395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:29:32.055233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:29:32.055233+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618333+0000","last_change":"2024-03-24T11:31:30.728130+0000","last_active":"2024-03-24T11:32:32.618333+0000","last_peered":"2024-03-24T11:32:32.618333+0000","last_clean":"2024-03-24T11:32:32.618333+0000","last_became_active":"2024-03-24T11:30:35.353946+0000","last_became_peered":"2024-03-24T11:30:35.353946+0000","last_unstale":"2024-03-24T11:32:32.618333+0000","last_undegraded":"2024-03-24T11:32:32.618333+0000","last_fullsized":"2024-03-24T11:32:32.618333+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:30.728004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.728004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:49.089471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":632,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.707039+0000","last_change":"2024-03-24T11:31:36.682644+0000","last_active":"2024-03-24T11:32:32.707039+0000","last_peered":"2024-03-24T11:32:32.707039+0000","last_clean":"2024-03-24T11:32:32.707039+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:32:32.707039+0000","last_undegraded":"2024-03-24T11:32:32.707039+0000","last_fullsized":"2024-03-24T11:32:32.707039+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:36.682542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.682542+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.160138+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":448,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704365+0000","last_change":"2024-03-24T11:32:31.071719+0000","last_active":"2024-03-24T11:32:32.704365+0000","last_peered":"2024-03-24T11:32:32.704365+0000","last_clean":"2024-03-24T11:32:32.704365+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:32:32.704365+0000","last_undegraded":"2024-03-24T11:32:32.704365+0000","last_fullsized":"2024-03-24T11:32:32.704365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:31.071621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:31.071621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.164745+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704669+0000","last_change":"2024-03-24T11:31:06.835419+0000","last_active":"2024-03-24T11:32:32.704669+0000","last_peered":"2024-03-24T11:32:32.704669+0000","last_clean":"2024-03-24T11:32:32.704669+0000","last_became_active":"2024-03-24T11:28:43.188012+0000","last_became_peered":"2024-03-24T11:28:43.188012+0000","last_unstale":"2024-03-24T11:32:32.704669+0000","last_undegraded":"2024-03-24T11:32:32.704669+0000","last_fullsized":"2024-03-24T11:32:32.704669+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:06.835322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.835322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"213'24","reported_seq":492,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618428+0000","last_change":"2024-03-24T11:30:13.928893+0000","last_active":"2024-03-24T11:32:32.618428+0000","last_peered":"2024-03-24T11:32:32.618428+0000","last_clean":"2024-03-24T11:32:32.618428+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:32:32.618428+0000","last_undegraded":"2024-03-24T11:32:32.618428+0000","last_fullsized":"2024-03-24T11:32:32.618428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:30:13.928734+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:30:13.928734+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618637+0000","last_change":"2024-03-24T11:31:37.636078+0000","last_active":"2024-03-24T11:32:32.618637+0000","last_peered":"2024-03-24T11:32:32.618637+0000","last_clean":"2024-03-24T11:32:32.618637+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:32:32.618637+0000","last_undegraded":"2024-03-24T11:32:32.618637+0000","last_fullsized":"2024-03-24T11:32:32.618637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:37.635705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.635705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.929525+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":468,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704852+0000","last_change":"2024-03-24T11:31:19.311796+0000","last_active":"2024-03-24T11:32:32.704852+0000","last_peered":"2024-03-24T11:32:32.704852+0000","last_clean":"2024-03-24T11:32:32.704852+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:32:32.704852+0000","last_undegraded":"2024-03-24T11:32:32.704852+0000","last_fullsized":"2024-03-24T11:32:32.704852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:19.311581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:31:19.311581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:36.626082+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":440,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649000+0000","last_change":"2024-03-24T11:32:32.077273+0000","last_active":"2024-03-24T11:32:32.649000+0000","last_peered":"2024-03-24T11:32:32.649000+0000","last_clean":"2024-03-24T11:32:32.649000+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:32:32.649000+0000","last_undegraded":"2024-03-24T11:32:32.649000+0000","last_fullsized":"2024-03-24T11:32:32.649000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.077150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.077150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:35.573524+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"213'15","reported_seq":489,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704965+0000","last_change":"2024-03-24T11:31:36.651472+0000","last_active":"2024-03-24T11:32:32.704965+0000","last_peered":"2024-03-24T11:32:32.704965+0000","last_clean":"2024-03-24T11:32:32.704965+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:32:32.704965+0000","last_undegraded":"2024-03-24T11:32:32.704965+0000","last_fullsized":"2024-03-24T11:32:32.704965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:36.651340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.651340+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:49.260143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":64,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648634+0000","last_change":"2024-03-24T11:30:35.353582+0000","last_active":"2024-03-24T11:32:32.648634+0000","last_peered":"2024-03-24T11:32:32.648634+0000","last_clean":"2024-03-24T11:32:32.648634+0000","last_became_active":"2024-03-24T11:30:35.353230+0000","last_became_peered":"2024-03-24T11:30:35.353230+0000","last_unstale":"2024-03-24T11:32:32.648634+0000","last_undegraded":"2024-03-24T11:32:32.648634+0000","last_fullsized":"2024-03-24T11:32:32.648634+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:23.813152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:29:23.813152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":436,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617894+0000","last_change":"2024-03-24T11:29:29.048318+0000","last_active":"2024-03-24T11:32:32.617894+0000","last_peered":"2024-03-24T11:32:32.617894+0000","last_clean":"2024-03-24T11:32:32.617894+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:32:32.617894+0000","last_undegraded":"2024-03-24T11:32:32.617894+0000","last_fullsized":"2024-03-24T11:32:32.617894+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:29.046315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:29:29.046315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":448,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.703800+0000","last_change":"2024-03-24T11:32:24.051941+0000","last_active":"2024-03-24T11:32:32.703800+0000","last_peered":"2024-03-24T11:32:32.703800+0000","last_clean":"2024-03-24T11:32:32.703800+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:32:32.703800+0000","last_undegraded":"2024-03-24T11:32:32.703800+0000","last_fullsized":"2024-03-24T11:32:32.703800+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.051777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.051777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.050306+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":601,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617605+0000","last_change":"2024-03-24T11:31:06.830322+0000","last_active":"2024-03-24T11:32:32.617605+0000","last_peered":"2024-03-24T11:32:32.617605+0000","last_clean":"2024-03-24T11:32:32.617605+0000","last_became_active":"2024-03-24T11:28:43.192919+0000","last_became_peered":"2024-03-24T11:28:43.192919+0000","last_unstale":"2024-03-24T11:32:32.617605+0000","last_undegraded":"2024-03-24T11:32:32.617605+0000","last_fullsized":"2024-03-24T11:32:32.617605+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:06.830158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.830158+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"213'19","reported_seq":471,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649234+0000","last_change":"2024-03-24T11:30:13.139116+0000","last_active":"2024-03-24T11:32:32.649234+0000","last_peered":"2024-03-24T11:32:32.649234+0000","last_clean":"2024-03-24T11:32:32.649234+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:32:32.649234+0000","last_undegraded":"2024-03-24T11:32:32.649234+0000","last_fullsized":"2024-03-24T11:32:32.649234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:30:13.138980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:13.138980+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":468,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705479+0000","last_change":"2024-03-24T11:31:40.668063+0000","last_active":"2024-03-24T11:32:32.705479+0000","last_peered":"2024-03-24T11:32:32.705479+0000","last_clean":"2024-03-24T11:32:32.705479+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:32:32.705479+0000","last_undegraded":"2024-03-24T11:32:32.705479+0000","last_fullsized":"2024-03-24T11:32:32.705479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:40.667849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:40.667849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:09.047430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617620+0000","last_change":"2024-03-24T11:32:23.732096+0000","last_active":"2024-03-24T11:32:32.617620+0000","last_peered":"2024-03-24T11:32:32.617620+0000","last_clean":"2024-03-24T11:32:32.617620+0000","last_became_active":"2024-03-24T11:30:35.353571+0000","last_became_peered":"2024-03-24T11:30:35.353571+0000","last_unstale":"2024-03-24T11:32:32.617620+0000","last_undegraded":"2024-03-24T11:32:32.617620+0000","last_fullsized":"2024-03-24T11:32:32.617620+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:23.731967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.731967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:38.504397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":440,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648655+0000","last_change":"2024-03-24T11:30:24.313618+0000","last_active":"2024-03-24T11:32:32.648655+0000","last_peered":"2024-03-24T11:32:32.648655+0000","last_clean":"2024-03-24T11:32:32.648655+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:32:32.648655+0000","last_undegraded":"2024-03-24T11:32:32.648655+0000","last_fullsized":"2024-03-24T11:32:32.648655+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:24.313152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.313152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"213'10","reported_seq":454,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617317+0000","last_change":"2024-03-24T11:29:30.032558+0000","last_active":"2024-03-24T11:32:32.617317+0000","last_peered":"2024-03-24T11:32:32.617317+0000","last_clean":"2024-03-24T11:32:32.617317+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:32:32.617317+0000","last_undegraded":"2024-03-24T11:32:32.617317+0000","last_fullsized":"2024-03-24T11:32:32.617317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:29:30.032476+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:29:30.032476+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704447+0000","last_change":"2024-03-24T11:30:08.448876+0000","last_active":"2024-03-24T11:32:32.704447+0000","last_peered":"2024-03-24T11:32:32.704447+0000","last_clean":"2024-03-24T11:32:32.704447+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:32:32.704447+0000","last_undegraded":"2024-03-24T11:32:32.704447+0000","last_fullsized":"2024-03-24T11:32:32.704447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:08.448741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:08.448741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":468,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706277+0000","last_change":"2024-03-24T11:31:22.909509+0000","last_active":"2024-03-24T11:32:32.706277+0000","last_peered":"2024-03-24T11:32:32.706277+0000","last_clean":"2024-03-24T11:32:32.706277+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:32:32.706277+0000","last_undegraded":"2024-03-24T11:32:32.706277+0000","last_fullsized":"2024-03-24T11:32:32.706277+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.909401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617841+0000","last_change":"2024-03-24T11:31:17.891282+0000","last_active":"2024-03-24T11:32:32.617841+0000","last_peered":"2024-03-24T11:32:32.617841+0000","last_clean":"2024-03-24T11:32:32.617841+0000","last_became_active":"2024-03-24T11:30:35.353704+0000","last_became_peered":"2024-03-24T11:30:35.353704+0000","last_unstale":"2024-03-24T11:32:32.617841+0000","last_undegraded":"2024-03-24T11:32:32.617841+0000","last_fullsized":"2024-03-24T11:32:32.617841+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:17.890440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:17.890440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"213'24","reported_seq":517,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705331+0000","last_change":"2024-03-24T11:32:30.897616+0000","last_active":"2024-03-24T11:32:32.705331+0000","last_peered":"2024-03-24T11:32:32.705331+0000","last_clean":"2024-03-24T11:32:32.705331+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:32:32.705331+0000","last_undegraded":"2024-03-24T11:32:32.705331+0000","last_fullsized":"2024-03-24T11:32:32.705331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:30.897405+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.897405+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.068479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":452,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617297+0000","last_change":"2024-03-24T11:32:30.073961+0000","last_active":"2024-03-24T11:32:32.617297+0000","last_peered":"2024-03-24T11:32:32.617297+0000","last_clean":"2024-03-24T11:32:32.617297+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:32:32.617297+0000","last_undegraded":"2024-03-24T11:32:32.617297+0000","last_fullsized":"2024-03-24T11:32:32.617297+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:30.073873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.073873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:52.383142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705313+0000","last_change":"2024-03-24T11:31:14.751432+0000","last_active":"2024-03-24T11:32:32.705313+0000","last_peered":"2024-03-24T11:32:32.705313+0000","last_clean":"2024-03-24T11:32:32.705313+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:32:32.705313+0000","last_undegraded":"2024-03-24T11:32:32.705313+0000","last_fullsized":"2024-03-24T11:32:32.705313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.751309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.751309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:43.031712+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":464,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705633+0000","last_change":"2024-03-24T11:31:41.853862+0000","last_active":"2024-03-24T11:32:32.705633+0000","last_peered":"2024-03-24T11:32:32.705633+0000","last_clean":"2024-03-24T11:32:32.705633+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:32:32.705633+0000","last_undegraded":"2024-03-24T11:32:32.705633+0000","last_fullsized":"2024-03-24T11:32:32.705633+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.853751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:04.343556+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"213'20","reported_seq":495,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704674+0000","last_change":"2024-03-24T11:32:25.904952+0000","last_active":"2024-03-24T11:32:32.704674+0000","last_peered":"2024-03-24T11:32:32.704674+0000","last_clean":"2024-03-24T11:32:32.704674+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:32:32.704674+0000","last_undegraded":"2024-03-24T11:32:32.704674+0000","last_fullsized":"2024-03-24T11:32:32.704674+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:25.904851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.904851+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.755461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":477,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704426+0000","last_change":"2024-03-24T11:31:37.672115+0000","last_active":"2024-03-24T11:32:32.704426+0000","last_peered":"2024-03-24T11:32:32.704426+0000","last_clean":"2024-03-24T11:32:32.704426+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:32:32.704426+0000","last_undegraded":"2024-03-24T11:32:32.704426+0000","last_fullsized":"2024-03-24T11:32:32.704426+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:37.671989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.671989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.288597+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":648,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705095+0000","last_change":"2024-03-24T11:31:14.791425+0000","last_active":"2024-03-24T11:32:32.705095+0000","last_peered":"2024-03-24T11:32:32.705095+0000","last_clean":"2024-03-24T11:32:32.705095+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:32:32.705095+0000","last_undegraded":"2024-03-24T11:32:32.705095+0000","last_fullsized":"2024-03-24T11:32:32.705095+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:31:14.791280+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:23:53.129429+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.791280+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:32.723356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":111,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704630+0000","last_change":"2024-03-24T11:30:35.355569+0000","last_active":"2024-03-24T11:32:32.704630+0000","last_peered":"2024-03-24T11:32:32.704630+0000","last_clean":"2024-03-24T11:32:32.704630+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:32:32.704630+0000","last_undegraded":"2024-03-24T11:32:32.704630+0000","last_fullsized":"2024-03-24T11:32:32.704630+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:30:22.784182+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:30:22.784182+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"213'18","reported_seq":472,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617778+0000","last_change":"2024-03-24T11:31:11.260142+0000","last_active":"2024-03-24T11:32:32.617778+0000","last_peered":"2024-03-24T11:32:32.617778+0000","last_clean":"2024-03-24T11:32:32.617778+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:32:32.617778+0000","last_undegraded":"2024-03-24T11:32:32.617778+0000","last_fullsized":"2024-03-24T11:32:32.617778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.259991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.259991+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:40.286574+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":452,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618281+0000","last_change":"2024-03-24T11:30:11.883846+0000","last_active":"2024-03-24T11:32:32.618281+0000","last_peered":"2024-03-24T11:32:32.618281+0000","last_clean":"2024-03-24T11:32:32.618281+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:32:32.618281+0000","last_undegraded":"2024-03-24T11:32:32.618281+0000","last_fullsized":"2024-03-24T11:32:32.618281+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:11.883599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.883599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":452,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618767+0000","last_change":"2024-03-24T11:30:23.939208+0000","last_active":"2024-03-24T11:32:32.618767+0000","last_peered":"2024-03-24T11:32:32.618767+0000","last_clean":"2024-03-24T11:32:32.618767+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:32:32.618767+0000","last_undegraded":"2024-03-24T11:32:32.618767+0000","last_fullsized":"2024-03-24T11:32:32.618767+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:23.939073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:30:23.939073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":433,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649141+0000","last_change":"2024-03-24T11:30:08.308986+0000","last_active":"2024-03-24T11:32:32.649141+0000","last_peered":"2024-03-24T11:32:32.649141+0000","last_clean":"2024-03-24T11:32:32.649141+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:32:32.649141+0000","last_undegraded":"2024-03-24T11:32:32.649141+0000","last_fullsized":"2024-03-24T11:32:32.649141+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:08.308860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:08.308860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"213'27","reported_seq":507,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617792+0000","last_change":"2024-03-24T11:30:09.868426+0000","last_active":"2024-03-24T11:32:32.617792+0000","last_peered":"2024-03-24T11:32:32.617792+0000","last_clean":"2024-03-24T11:32:32.617792+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:32:32.617792+0000","last_undegraded":"2024-03-24T11:32:32.617792+0000","last_fullsized":"2024-03-24T11:32:32.617792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:30:09.868130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:09.868130+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705214+0000","last_change":"2024-03-24T11:31:10.852618+0000","last_active":"2024-03-24T11:32:32.705214+0000","last_peered":"2024-03-24T11:32:32.705214+0000","last_clean":"2024-03-24T11:32:32.705214+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:32:32.705214+0000","last_undegraded":"2024-03-24T11:32:32.705214+0000","last_fullsized":"2024-03-24T11:32:32.705214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:10.852490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.852490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.705572+0000","last_change":"2024-03-24T11:31:23.373859+0000","last_active":"2024-03-24T11:32:32.705572+0000","last_peered":"2024-03-24T11:32:32.705572+0000","last_clean":"2024-03-24T11:32:32.705572+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:32:32.705572+0000","last_undegraded":"2024-03-24T11:32:32.705572+0000","last_fullsized":"2024-03-24T11:32:32.705572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:23.373583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:31:23.373583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:39.855446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":449,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.619212+0000","last_change":"2024-03-24T11:31:09.866310+0000","last_active":"2024-03-24T11:32:32.619212+0000","last_peered":"2024-03-24T11:32:32.619212+0000","last_clean":"2024-03-24T11:32:32.619212+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:32:32.619212+0000","last_undegraded":"2024-03-24T11:32:32.619212+0000","last_fullsized":"2024-03-24T11:32:32.619212+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:09.866232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:31:09.866232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"213'7","reported_seq":469,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618097+0000","last_change":"2024-03-24T11:31:27.652238+0000","last_active":"2024-03-24T11:32:32.618097+0000","last_peered":"2024-03-24T11:32:32.618097+0000","last_clean":"2024-03-24T11:32:32.618097+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:32:32.618097+0000","last_undegraded":"2024-03-24T11:32:32.618097+0000","last_fullsized":"2024-03-24T11:32:32.618097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:31:27.652037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.652037+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:36.191792+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704895+0000","last_change":"2024-03-24T11:30:14.299035+0000","last_active":"2024-03-24T11:32:32.704895+0000","last_peered":"2024-03-24T11:32:32.704895+0000","last_clean":"2024-03-24T11:32:32.704895+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:32:32.704895+0000","last_undegraded":"2024-03-24T11:32:32.704895+0000","last_fullsized":"2024-03-24T11:32:32.704895+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:14.298903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.298903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":72,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618457+0000","last_change":"2024-03-24T11:31:41.715026+0000","last_active":"2024-03-24T11:32:32.618457+0000","last_peered":"2024-03-24T11:32:32.618457+0000","last_clean":"2024-03-24T11:32:32.618457+0000","last_became_active":"2024-03-24T11:30:35.359239+0000","last_became_peered":"2024-03-24T11:30:35.359239+0000","last_unstale":"2024-03-24T11:32:32.618457+0000","last_undegraded":"2024-03-24T11:32:32.618457+0000","last_fullsized":"2024-03-24T11:32:32.618457+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:11.406185+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":448,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706522+0000","last_change":"2024-03-24T11:30:17.465431+0000","last_active":"2024-03-24T11:32:32.706522+0000","last_peered":"2024-03-24T11:32:32.706522+0000","last_clean":"2024-03-24T11:32:32.706522+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:32:32.706522+0000","last_undegraded":"2024-03-24T11:32:32.706522+0000","last_fullsized":"2024-03-24T11:32:32.706522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.465226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.465226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"213'15","reported_seq":473,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.618253+0000","last_change":"2024-03-24T11:30:15.882584+0000","last_active":"2024-03-24T11:32:32.618253+0000","last_peered":"2024-03-24T11:32:32.618253+0000","last_clean":"2024-03-24T11:32:32.618253+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:32:32.618253+0000","last_undegraded":"2024-03-24T11:32:32.618253+0000","last_fullsized":"2024-03-24T11:32:32.618253+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:15.882446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:15.882446+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":468,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706963+0000","last_change":"2024-03-24T11:31:41.687584+0000","last_active":"2024-03-24T11:32:32.706963+0000","last_peered":"2024-03-24T11:32:32.706963+0000","last_clean":"2024-03-24T11:32:32.706963+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:32:32.706963+0000","last_undegraded":"2024-03-24T11:32:32.706963+0000","last_fullsized":"2024-03-24T11:32:32.706963+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.687474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.687474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:06.652087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.706009+0000","last_change":"2024-03-24T11:31:15.474018+0000","last_active":"2024-03-24T11:32:32.706009+0000","last_peered":"2024-03-24T11:32:32.706009+0000","last_clean":"2024-03-24T11:32:32.706009+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:32:32.706009+0000","last_undegraded":"2024-03-24T11:32:32.706009+0000","last_fullsized":"2024-03-24T11:32:32.706009+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.473909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.473909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":448,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704083+0000","last_change":"2024-03-24T11:31:18.759423+0000","last_active":"2024-03-24T11:32:32.704083+0000","last_peered":"2024-03-24T11:32:32.704083+0000","last_clean":"2024-03-24T11:32:32.704083+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:32:32.704083+0000","last_undegraded":"2024-03-24T11:32:32.704083+0000","last_fullsized":"2024-03-24T11:32:32.704083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:18.759165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.759165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"213'27","reported_seq":483,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.649297+0000","last_change":"2024-03-24T11:31:41.107651+0000","last_active":"2024-03-24T11:32:32.649297+0000","last_peered":"2024-03-24T11:32:32.649297+0000","last_clean":"2024-03-24T11:32:32.649297+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:32:32.649297+0000","last_undegraded":"2024-03-24T11:32:32.649297+0000","last_fullsized":"2024-03-24T11:32:32.649297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:31:41.107512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.107512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:44.858057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":445,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617846+0000","last_change":"2024-03-24T11:30:23.137546+0000","last_active":"2024-03-24T11:32:32.617846+0000","last_peered":"2024-03-24T11:32:32.617846+0000","last_clean":"2024-03-24T11:32:32.617846+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:32:32.617846+0000","last_undegraded":"2024-03-24T11:32:32.617846+0000","last_fullsized":"2024-03-24T11:32:32.617846+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:23.137393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:23.137393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":452,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617799+0000","last_change":"2024-03-24T11:32:24.692923+0000","last_active":"2024-03-24T11:32:32.617799+0000","last_peered":"2024-03-24T11:32:32.617799+0000","last_clean":"2024-03-24T11:32:32.617799+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:32:32.617799+0000","last_undegraded":"2024-03-24T11:32:32.617799+0000","last_fullsized":"2024-03-24T11:32:32.617799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.692844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.692844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.656723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":432,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.648507+0000","last_change":"2024-03-24T11:31:14.327893+0000","last_active":"2024-03-24T11:32:32.648507+0000","last_peered":"2024-03-24T11:32:32.648507+0000","last_clean":"2024-03-24T11:32:32.648507+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:32:32.648507+0000","last_undegraded":"2024-03-24T11:32:32.648507+0000","last_fullsized":"2024-03-24T11:32:32.648507+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.327777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.327777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"213'15","reported_seq":497,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617685+0000","last_change":"2024-03-24T11:31:08.802687+0000","last_active":"2024-03-24T11:32:32.617685+0000","last_peered":"2024-03-24T11:32:32.617685+0000","last_clean":"2024-03-24T11:32:32.617685+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:32:32.617685+0000","last_undegraded":"2024-03-24T11:32:32.617685+0000","last_fullsized":"2024-03-24T11:32:32.617685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:08.802538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.802538+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704251+0000","last_change":"2024-03-24T11:29:39.487723+0000","last_active":"2024-03-24T11:32:32.704251+0000","last_peered":"2024-03-24T11:32:32.704251+0000","last_clean":"2024-03-24T11:32:32.704251+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:32:32.704251+0000","last_undegraded":"2024-03-24T11:32:32.704251+0000","last_fullsized":"2024-03-24T11:32:32.704251+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:39.487609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:29:39.487609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":468,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.704307+0000","last_change":"2024-03-24T11:32:28.712422+0000","last_active":"2024-03-24T11:32:32.704307+0000","last_peered":"2024-03-24T11:32:32.704307+0000","last_clean":"2024-03-24T11:32:32.704307+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:32:32.704307+0000","last_undegraded":"2024-03-24T11:32:32.704307+0000","last_fullsized":"2024-03-24T11:32:32.704307+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.712294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.712294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.891340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":448,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-24T11:32:32.617945+0000","last_change":"2024-03-24T11:31:07.885306+0000","last_active":"2024-03-24T11:32:32.617945+0000","last_peered":"2024-03-24T11:32:32.617945+0000","last_clean":"2024-03-24T11:32:32.617945+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:32:32.617945+0000","last_undegraded":"2024-03-24T11:32:32.617945+0000","last_fullsized":"2024-03-24T11:32:32.617945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:07.885177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:07.885177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267170,"num_write_kb":8549377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73144,"ondisk_log_size":73144,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":120,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070632,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491396,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253756,"statfs":{"total":95995035648,"available":95491846144,"internally_reserved":0,"allocated":716800,"data_stored":427739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":502462923},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364973,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333572,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93411580,"statfs":{"total":95995035648,"available":95653457920,"internally_reserved":0,"allocated":1044480,"data_stored":763774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":340392390},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":497008,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93248144,"statfs":{"total":95995035648,"available":95486099456,"internally_reserved":0,"allocated":757760,"data_stored":436931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415388,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329764,"statfs":{"total":95995035648,"available":95569678336,"internally_reserved":0,"allocated":1064960,"data_stored":763308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411522,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334804,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410348,"statfs":{"total":95995035648,"available":95652196352,"internally_reserved":0,"allocated":991232,"data_stored":698077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149883,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496300,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":93248852,"statfs":{"total":95995035648,"available":95486824448,"internally_reserved":0,"allocated":1347584,"data_stored":1026491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":506722759},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986360,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320668,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93424484,"statfs":{"total":95995035648,"available":95666671616,"internally_reserved":0,"allocated":741376,"data_stored":436301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659314,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491664,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253488,"statfs":{"total":95995035648,"available":95491571712,"internally_reserved":0,"allocated":987136,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:32:40.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:32:40.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:40.843 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 5.1 deep-scrub starts 2024-03-24T11:32:40.843 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 5.1 deep-scrub ok 2024-03-24T11:32:40.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:32:40.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:32:40.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:32:40.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:32:40.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 1.14 scrub starts 2024-03-24T11:32:40.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:40 smithi181 ceph-mon[33998]: 1.14 scrub ok 2024-03-24T11:32:40.848 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:40.950 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:40.974 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:40.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:41.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:41.100 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:41.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:41.226 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:41.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:41.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:41.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:41.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:41.602 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:41.663 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_2' removed 2024-03-24T11:32:41.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:41.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:32:41.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:32:41.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:32:41.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: pgmap v578: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:32:41.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:32:41.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:32:41.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:32:41.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:32:41.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3556927502' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:32:41.707 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:41 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:32:41.729 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:41.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:41.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:32:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:32:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:32:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: pgmap v578: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:32:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:32:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:32:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3556927502' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: pgmap v578: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3556927502' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:32:41.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:41 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:32:41.971 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:41.994 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:41.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:42.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:42.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:42.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:42.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:42.249 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:42.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:42.313 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:42.341 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T11:32:42.342 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T11:32:42.342 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-24T11:32:42.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-24T11:32:42.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:42.351 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126625 7.1 7:8279287a:::benchmark_data_smithi179_78267_object7431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.576293+0000", 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 597.77275996000003, 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.108871729, 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:42.352 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126625 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576293+0000", 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576293+0000", 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576295+0000", 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.559e-06 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576300+0000", 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8320000000000002e-06 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576313+0000", 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3052e-05 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576322+0000", 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8519999999999993e-06 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576397+0000", 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5067000000000005e-05 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576510+0000", 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112689 2024-03-24T11:32:42.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.576588+0000", 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7998999999999996e-05 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.579000+0000", 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024118529999999998 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.579200+0000", 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019967300000000001 2024-03-24T11:32:42.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685057+0000", 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105857224 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685118+0000", 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0671999999999998e-05 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.685165+0000", 2024-03-24T11:32:42.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7256999999999998e-05 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126629 7.1 7:86ceb58f:::benchmark_data_smithi179_78267_object7435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.580293+0000", 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 597.76875979099998, 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113232627, 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:42.358 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126629 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580293+0000", 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580293+0000", 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580294+0000", 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1680000000000001e-06 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580299+0000", 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9089999999999999e-06 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580313+0000", 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3719e-05 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580322+0000", 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8130000000000002e-06 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580382+0000", 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0529999999999998e-05 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580491+0000", 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010838800000000001 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.580606+0000", 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115586 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.583275+0000", 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026691449999999999 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.583615+0000", 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033952800000000002 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.693457+0000", 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109841828 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.693496+0000", 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8744e-05 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.693526+0000", 2024-03-24T11:32:42.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0269000000000001e-05 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126636 7.1 7:8ae9c76e:::benchmark_data_smithi179_78267_object7442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.586631+0000", 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 597.76242151999998, 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11141478, 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.364 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126636 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586631+0000", 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586631+0000", 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586633+0000", 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.314e-06 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586638+0000", 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9169999999999997e-06 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586651+0000", 2024-03-24T11:32:42.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3788000000000001e-05 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586660+0000", 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.833e-06 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586728+0000", 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7809000000000001e-05 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586845+0000", 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011691700000000001 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.586957+0000", 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000111525 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.589474+0000", 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025169350000000001 2024-03-24T11:32:42.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.589828+0000", 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00035489099999999997 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.697927+0000", 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.108098507 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.698001+0000", 2024-03-24T11:32:42.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3652999999999994e-05 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.698046+0000", 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5691e-05 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:126642 7.1 7:88145bcf:::benchmark_data_smithi179_78267_object7448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:22:44.591629+0000", 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 597.75742401000002, 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11300893300000001, 2024-03-24T11:32:42.370 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 126642 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591629+0000", 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591629+0000", 2024-03-24T11:32:42.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591630+0000", 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0869999999999999e-06 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591636+0000", 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9859999999999999e-06 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591648+0000", 2024-03-24T11:32:42.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2163e-05 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591657+0000", 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3450000000000006e-06 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591706+0000", 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9165000000000001e-05 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591838+0000", 2024-03-24T11:32:42.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000132106 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.591963+0000", 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000124948 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.594631+0000", 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026683739999999998 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.595208+0000", 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057702199999999995 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.704533+0000", 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10932528699999999 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.704587+0000", 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3393999999999997e-05 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:22:44.704638+0000", 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1056e-05 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:199460 7.1 7:87ede80d:::benchmark_data_smithi179_78267_object80266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:01.740574+0000", 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 520.60847925600001, 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113635852, 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.376 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199460 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740574+0000", 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740574+0000", 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740575+0000", 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.209e-06 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740579+0000", 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9190000000000001e-06 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740591+0000", 2024-03-24T11:32:42.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1914e-05 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740598+0000", 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7689999999999999e-06 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740665+0000", 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7747e-05 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740765+0000", 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100078 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.740838+0000", 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2358000000000001e-05 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.748221+0000", 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073835150000000002 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.749807+0000", 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001585691 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.854136+0000", 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10432912599999999 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.854172+0000", 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6298999999999999e-05 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.854210+0000", 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7227000000000001e-05 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:199462 7.1 7:87bec4d9:::benchmark_data_smithi179_78267_object80268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:01.743228+0000", 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 520.60582476499997, 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112712804, 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.383 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199462 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.743228+0000", 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.743228+0000", 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.743229+0000", 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.159e-06 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.743234+0000", 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4649999999999996e-06 2024-03-24T11:32:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.743246+0000", 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1717e-05 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.743253+0000", 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2609999999999996e-06 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.745395+0000", 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002142662 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.745506+0000", 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110172 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.745613+0000", 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107494 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.748330+0000", 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027173029999999999 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.749903+0000", 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001572312 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.855832+0000", 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10592945600000001 2024-03-24T11:32:42.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.855891+0000", 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8671000000000001e-05 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:01.855941+0000", 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0132e-05 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:15261 8.2 8:4ff01512:::benchmark_data_smithi179_105658_object15260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:25:19.249347+0000", 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 443.099705986, 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.089156840000000001, 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 15261 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.390 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249347+0000", 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249347+0000", 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249349+0000", 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9609999999999999e-06 2024-03-24T11:32:42.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249415+0000", 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6327000000000004e-05 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249428+0000", 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.234e-05 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249436+0000", 2024-03-24T11:32:42.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.909e-06 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249499+0000", 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3200000000000005e-05 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249649+0000", 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014978300000000001 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.249797+0000", 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000148228 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.250805+0000", 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010082229999999999 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.251885+0000", 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010802190000000001 2024-03-24T11:32:42.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.338399+0000", 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.086514069999999998 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.338446+0000", 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6697999999999999e-05 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:25:19.338504+0000", 2024-03-24T11:32:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7881999999999999e-05 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:113241 8.2 8:41dd97b5:::benchmark_data_smithi179_105658_object113240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.148381+0000", 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 373.20067173199999, 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094579893999999998, 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.396 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113241 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148381+0000", 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148381+0000", 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148391+0000", 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0128e-05 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148445+0000", 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3378999999999998e-05 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148460+0000", 2024-03-24T11:32:42.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5332000000000002e-05 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148469+0000", 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1449999999999997e-06 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148528+0000", 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8607999999999999e-05 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148623+0000", 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4845000000000004e-05 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148711+0000", 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8790999999999997e-05 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.155066+0000", 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063544090000000001 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.162393+0000", 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073266149999999999 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242697+0000", 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.080304747999999995 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242791+0000", 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3332999999999996e-05 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.242961+0000", 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017056100000000001 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:113243 8.2 8:4c9bdbcd:::benchmark_data_smithi179_105658_object113242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.148495+0000", 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 373.200557635, 2024-03-24T11:32:42.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094642911999999996, 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113243 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148495+0000", 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148495+0000", 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148496+0000", 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3499999999999996e-07 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148525+0000", 2024-03-24T11:32:42.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8889000000000001e-05 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148532+0000", 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5259999999999996e-06 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.148538+0000", 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.412e-06 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.151299+0000", 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027612579999999999 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.151425+0000", 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126217 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.151854+0000", 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042914400000000001 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.155299+0000", 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034444509999999998 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.164610+0000", 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0093116540000000008 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.243024+0000", 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078413246000000006 2024-03-24T11:32:42.407 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.243093+0000", 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9258000000000007e-05 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.243138+0000", 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5321999999999997e-05 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.408 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:113249 8.2 8:48db2c3e:::benchmark_data_smithi179_105658_object113248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.155939+0000", 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 373.19311397799999, 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087352473, 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113249 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.155939+0000", 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.155939+0000", 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.155940+0000", 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.229e-06 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156026+0000", 2024-03-24T11:32:42.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5772000000000005e-05 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156037+0000", 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0559e-05 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156044+0000", 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2969999999999998e-06 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156088+0000", 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4060000000000002e-05 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156216+0000", 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012799900000000001 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.156366+0000", 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015021699999999999 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.160680+0000", 2024-03-24T11:32:42.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043134690000000003 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.164642+0000", 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039620159999999996 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.243193+0000", 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078551371999999994 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.243248+0000", 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5052000000000001e-05 2024-03-24T11:32:42.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.243291+0000", 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3430999999999998e-05 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:113254 8.2 8:44631bd0:::benchmark_data_smithi179_105658_object113253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:29.158182+0000", 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 373.19087093299999, 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087087654, 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.414 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113254 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158182+0000", 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158182+0000", 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158184+0000", 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.446e-06 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158244+0000", 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0386000000000001e-05 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158257+0000", 2024-03-24T11:32:42.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3271999999999999e-05 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158266+0000", 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5650000000000002e-06 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158469+0000", 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000203189 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158605+0000", 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013657 2024-03-24T11:32:42.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.158764+0000", 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000158636 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.160816+0000", 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020516620000000001 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.164736+0000", 2024-03-24T11:32:42.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039203880000000003 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.245157+0000", 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.080420866999999993 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.245229+0000", 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1727e-05 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:29.245270+0000", 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0945999999999997e-05 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:137376 8.2 8:45e47d05:::benchmark_data_smithi179_105658_object9334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:52.714512+0000", 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 349.63454130399998, 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.089078523000000007, 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.420 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137376 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.714512+0000", 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.714512+0000", 2024-03-24T11:32:42.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.714513+0000", 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7999999999999993e-07 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.714517+0000", 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7090000000000001e-06 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.714529+0000", 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1742999999999999e-05 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.714537+0000", 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3810000000000001e-06 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.716431+0000", 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001894153 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.716535+0000", 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010438900000000001 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.716641+0000", 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105956 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.719085+0000", 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024436839999999998 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.724740+0000", 2024-03-24T11:32:42.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056556039999999998 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.803435+0000", 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.078694875999999997 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.803538+0000", 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010248599999999999 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.425 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.803590+0000", 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2562000000000003e-05 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:137388 8.2 8:425b7036:::benchmark_data_smithi179_105658_object9346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:26:52.719970+0000", 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 349.62908323400001, 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.086932668000000005, 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.426 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137388 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.719970+0000", 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.719970+0000", 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.719971+0000", 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5799999999999999e-06 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.719976+0000", 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1590000000000001e-06 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.719988+0000", 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1684e-05 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.719996+0000", 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6420000000000007e-06 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.720047+0000", 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0676000000000001e-05 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.720157+0000", 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011034700000000001 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.720264+0000", 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010746100000000001 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.722178+0000", 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001913945 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.724771+0000", 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002593204 2024-03-24T11:32:42.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.806759+0000", 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.081987956000000001 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.806856+0000", 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6910000000000006e-05 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:26:52.806902+0000", 2024-03-24T11:32:42.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6103999999999998e-05 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:164409 8.2 8:48d021ac:::benchmark_data_smithi179_105658_object36367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.314467+0000", 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 327.034586031, 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11249822900000001, 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.432 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164409 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314467+0000", 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314467+0000", 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314468+0000", 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5290000000000001e-06 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314474+0000", 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.626e-06 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314488+0000", 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3807e-05 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314496+0000", 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.746e-06 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314652+0000", 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155845 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314793+0000", 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014123200000000001 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314912+0000", 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119392 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.317131+0000", 2024-03-24T11:32:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022192370000000002 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.317276+0000", 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000144361 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.426809+0000", 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109533634 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.426913+0000", 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010338 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.426965+0000", 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2439999999999999e-05 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:164995 8.2 8:49681ed3:::benchmark_data_smithi179_105658_object36953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.930195+0000", 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.41885783200001, 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13266282800000001, 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164995 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930195+0000", 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930195+0000", 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930196+0000", 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.066e-06 2024-03-24T11:32:42.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930201+0000", 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2749999999999997e-06 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930211+0000", 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0447e-05 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930219+0000", 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8199999999999997e-06 2024-03-24T11:32:42.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930435+0000", 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021634900000000001 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930554+0000", 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011854 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930671+0000", 2024-03-24T11:32:42.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011717700000000001 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.940381+0000", 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097098840000000002 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.940856+0000", 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047517399999999997 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.062665+0000", 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12180882899999999 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.062742+0000", 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7071999999999997e-05 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.062858+0000", 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011619499999999999 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.444 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:164996 8.2 8:4a9be0e9:::benchmark_data_smithi179_105658_object36954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.930325+0000", 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.41872809199998, 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13268659299999999, 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.445 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164996 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930325+0000", 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930325+0000", 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930326+0000", 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0699999999999996e-07 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930330+0000", 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5990000000000004e-06 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930342+0000", 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2091e-05 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930351+0000", 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2490000000000001e-06 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933053+0000", 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027027240000000001 2024-03-24T11:32:42.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933167+0000", 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114195 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933290+0000", 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122655 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.940401+0000", 2024-03-24T11:32:42.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071104410000000003 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.940959+0000", 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00055884999999999995 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.062915+0000", 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.121955143 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.062970+0000", 2024-03-24T11:32:42.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5362999999999998e-05 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.063011+0000", 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1576000000000003e-05 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:164997 8.2 8:461ea0f8:::benchmark_data_smithi179_105658_object36955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.930363+0000", 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.418689723, 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.132765513, 2024-03-24T11:32:42.452 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164997 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930363+0000", 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930363+0000", 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930364+0000", 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4700000000000001e-07 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930367+0000", 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7700000000000002e-06 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930374+0000", 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2230000000000001e-06 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.930380+0000", 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7860000000000002e-06 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935700+0000", 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053206690000000001 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935831+0000", 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013085399999999999 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935950+0000", 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011856 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.940406+0000", 2024-03-24T11:32:42.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044561729999999999 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.941041+0000", 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00063524499999999999 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.063058+0000", 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.122016598 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.063091+0000", 2024-03-24T11:32:42.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3390999999999999e-05 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.063129+0000", 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7697000000000002e-05 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:165004 8.2 8:443785db:::benchmark_data_smithi179_105658_object36962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.935321+0000", 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.41373155899998, 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12796286300000001, 2024-03-24T11:32:42.458 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165004 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935321+0000", 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935321+0000", 2024-03-24T11:32:42.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935323+0000", 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3790000000000001e-06 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935328+0000", 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0560000000000004e-06 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935345+0000", 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7232e-05 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.935354+0000", 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2760000000000006e-06 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.938236+0000", 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028818379999999998 2024-03-24T11:32:42.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.938367+0000", 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000131259 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.938484+0000", 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011663900000000001 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.940468+0000", 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019837779999999998 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.941089+0000", 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062094600000000004 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.063181+0000", 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12209170699999999 2024-03-24T11:32:42.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.063239+0000", 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8090000000000001e-05 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.063284+0000", 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5663e-05 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:165183 8.2 8:4ca49cc4:::benchmark_data_smithi179_105658_object37141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:16.171880+0000", 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 326.177173353, 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.091571361000000004, 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165183 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.171880+0000", 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.171880+0000", 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.171881+0000", 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5379999999999999e-06 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.171887+0000", 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5389999999999998e-06 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.171900+0000", 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2798e-05 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.171907+0000", 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8429999999999993e-06 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.171981+0000", 2024-03-24T11:32:42.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3192e-05 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.172090+0000", 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109487 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.172194+0000", 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103766 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.175332+0000", 2024-03-24T11:32:42.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031386859999999999 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.177341+0000", 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020086819999999999 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.263383+0000", 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.086042274000000002 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.263420+0000", 2024-03-24T11:32:42.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6508999999999999e-05 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.263451+0000", 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1047000000000003e-05 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:218854 9.e 9:72f19eb7:::benchmark_data_smithi179_130919_object85267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:44.704541+0000", 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.644512169999999, 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072553570999999997, 2024-03-24T11:32:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 218854 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704541+0000", 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704541+0000", 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:42.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704542+0000", 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0499999999999999e-06 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704545+0000", 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0960000000000001e-06 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704554+0000", 2024-03-24T11:32:42.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5839999999999995e-06 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704560+0000", 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1210000000000003e-06 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704835+0000", 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027575899999999999 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:42.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.704969+0000", 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000133619 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,7", 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.705091+0000", 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012228200000000001 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.707515+0000", 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024234700000000001 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.708080+0000", 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056538499999999995 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.776990+0000", 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.068910017000000004 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.777046+0000", 2024-03-24T11:32:42.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5398000000000001e-05 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:44.777094+0000", 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8789999999999999e-05 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:42.476 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:32:42.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:42.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:42.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:42.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:42.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:42.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:42.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:42.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:42.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:42.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:42.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:42.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:42.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:42.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[29608]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[29608]: 5.1f deep-scrub starts 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[29608]: 5.1f deep-scrub ok 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[37907]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[37907]: 5.1f deep-scrub starts 2024-03-24T11:32:42.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:42 smithi179 ceph-mon[37907]: 5.1f deep-scrub ok 2024-03-24T11:32:42.978 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:42.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:43.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:42 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:32:43.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:42 smithi181 ceph-mon[33998]: osdmap e265: 8 total, 8 up, 8 in 2024-03-24T11:32:43.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:42 smithi181 ceph-mon[33998]: 5.1f deep-scrub starts 2024-03-24T11:32:43.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:42 smithi181 ceph-mon[33998]: 5.1f deep-scrub ok 2024-03-24T11:32:43.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:43.104 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:43.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:43.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:43.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:43.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:43.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:32:43.331 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:43.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:43.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:43.483 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:43.507 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:43.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:43.556 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:43.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:43.632 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:43.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:43.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:43.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:43.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:43.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:43.884 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:43.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:43.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 5.3 deep-scrub starts 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 5.3 deep-scrub ok 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: pgmap v580: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:32:43.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:43 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 5.3 deep-scrub starts 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 5.3 deep-scrub ok 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: pgmap v580: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:32:43.912 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 5.3 deep-scrub starts 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 5.3 deep-scrub ok 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: pgmap v580: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:32:43.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:43 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:32:43.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:44.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:44.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:44.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:44.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:44.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:44.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:44.387 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:44.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:44.512 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:44.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:44.638 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:32:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.639 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:44.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:44.657 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:32:44.677 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:32:44.700 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:32:44.726 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:32:44.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:44.748 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:32:44.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:44.767 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:32:44.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3785802205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:32:44.769 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:32:44.770 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3785802205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:44.788 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:32:44.789 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:32:44.789 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:32:44.789 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:32:44.789 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:32:44.789 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:32:44.789 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:32:44.789 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:32:44.790 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:32:44.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:44.890 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:44.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:44.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:32:44.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3785802205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:44.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.016 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:45.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:45.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:45.141 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:45.231 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:32:45.231 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:32:45.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:45.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:45.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:45.392 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:45.460 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:32:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:32:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:32:45.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:45.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:45.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:45.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:45.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.770 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:45.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:45.815 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:45 smithi181 ceph-mon[33998]: pgmap v581: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:32:45.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:32:45.866 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:45 smithi179 ceph-mon[29608]: pgmap v581: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:32:45.866 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:45 smithi179 ceph-mon[37907]: pgmap v581: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:32:45.871 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:45.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:45.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:46.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:46.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:46.087 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:46.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:46.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:46.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:46.286 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:46.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:46.412 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:46.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:46.538 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:46.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:46.663 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:46.678 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:32:46.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:46.782 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:32:46.782 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:32:46.782 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:32:46.782 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:32:46.782 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1352359153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:32:46.783 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:32:46.783 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:32:46.783 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:32:46.783 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:32:46.783 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1352359153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:32:46.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:46.889 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:46.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:46 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:32:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:46 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:32:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:46 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:32:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:46 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:32:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1352359153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:32:46.913 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:46.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:47.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:47.015 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:47.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:47.140 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:47.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:47.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:47.288 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:47.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:47.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:47.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:47.540 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:47.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:47.665 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:47.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:47.773 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[29608]: 3.12 scrub starts 2024-03-24T11:32:47.773 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[29608]: 3.12 scrub ok 2024-03-24T11:32:47.773 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[29608]: pgmap v582: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:32:47.773 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:47.773 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1352359153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:32:47.773 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[29608]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T11:32:47.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[37907]: 3.12 scrub starts 2024-03-24T11:32:47.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[37907]: 3.12 scrub ok 2024-03-24T11:32:47.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[37907]: pgmap v582: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:32:47.774 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:47.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1352359153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:32:47.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:47 smithi179 ceph-mon[37907]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T11:32:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.794 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:47.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:47.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:47 smithi181 ceph-mon[33998]: 3.12 scrub starts 2024-03-24T11:32:47.850 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:47 smithi181 ceph-mon[33998]: 3.12 scrub ok 2024-03-24T11:32:47.850 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:47 smithi181 ceph-mon[33998]: pgmap v582: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-24T11:32:47.850 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:47 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:47.850 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1352359153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:32:47.850 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:47 smithi181 ceph-mon[33998]: osdmap e266: 8 total, 8 up, 8 in 2024-03-24T11:32:47.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:47.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:47.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:47.976 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:32:47.976 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:32:48.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:48.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:48.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:48.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:48.207 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:32:48.210 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:32:48.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:48.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:48.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:48.420 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:48.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:48.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:48.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.675 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:48.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:48.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:32:48.777 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:48.800 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:48.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:32:48.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:48.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:48.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:48.971 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:49.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:49.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:49.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:49.183 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:49.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:49.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:49.345 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:49 smithi181 ceph-mon[33998]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:32:49.345 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:49 smithi181 ceph-mon[33998]: pgmap v584: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:32:49.346 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:49 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:49.346 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:49.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:49.437 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:49.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:49.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:32:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.564 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:49.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[37907]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:32:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[37907]: pgmap v584: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:32:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:49.587 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[29608]: from='client.25457 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:32:49.587 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[29608]: pgmap v584: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T11:32:49.587 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:49.587 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:32:49.666 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:49.694 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:49.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:49.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:49.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:49.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:49.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:50.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:50.075 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:50.148 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2024-03-24T11:32:46.676978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1285,"num_read_kb":1037,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3512,"ondisk_log_size":3512,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":480,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3386328,"kb_used_data":7612,"kb_used_omap":12,"kb_used_meta":3378611,"kb_avail":746574888,"statfs":{"total":767960285184,"available":764492685312,"internally_reserved":0,"allocated":7794688,"data_stored":5324234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3459698225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":82,"apply_latency_ms":82,"commit_latency_ns":82000000,"apply_latency_ns":82000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-267170,"num_write_kb":-8549377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.376689"},"pg_stats":[{"pgid":"4.c","version":"213'20","reported_seq":489,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.644301+0000","last_change":"2024-03-24T11:32:36.799202+0000","last_active":"2024-03-24T11:32:41.644301+0000","last_peered":"2024-03-24T11:32:41.644301+0000","last_clean":"2024-03-24T11:32:41.644301+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:32:41.644301+0000","last_undegraded":"2024-03-24T11:32:41.644301+0000","last_fullsized":"2024-03-24T11:32:41.644301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:36.799084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.799084+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:06.179015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646292+0000","last_change":"2024-03-24T11:29:27.048143+0000","last_active":"2024-03-24T11:32:41.646292+0000","last_peered":"2024-03-24T11:32:41.646292+0000","last_clean":"2024-03-24T11:32:41.646292+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:32:41.646292+0000","last_undegraded":"2024-03-24T11:32:41.646292+0000","last_fullsized":"2024-03-24T11:32:41.646292+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:27.048023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:29:27.048023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":653,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643410+0000","last_change":"2024-03-24T11:32:37.091705+0000","last_active":"2024-03-24T11:32:41.643410+0000","last_peered":"2024-03-24T11:32:41.643410+0000","last_clean":"2024-03-24T11:32:41.643410+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:32:41.643410+0000","last_undegraded":"2024-03-24T11:32:41.643410+0000","last_fullsized":"2024-03-24T11:32:41.643410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:37.091630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.091630+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.785943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.644122+0000","last_change":"2024-03-24T11:31:15.842164+0000","last_active":"2024-03-24T11:32:41.644122+0000","last_peered":"2024-03-24T11:32:41.644122+0000","last_clean":"2024-03-24T11:32:41.644122+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:32:41.644122+0000","last_undegraded":"2024-03-24T11:32:41.644122+0000","last_fullsized":"2024-03-24T11:32:41.644122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.842054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.842054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":508,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643680+0000","last_change":"2024-03-24T11:31:10.289542+0000","last_active":"2024-03-24T11:32:41.643680+0000","last_peered":"2024-03-24T11:32:41.643680+0000","last_clean":"2024-03-24T11:32:41.643680+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:32:41.643680+0000","last_undegraded":"2024-03-24T11:32:41.643680+0000","last_fullsized":"2024-03-24T11:32:41.643680+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:10.289309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.289309+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"213'39","reported_seq":548,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.683810+0000","last_change":"2024-03-24T11:32:42.683845+0000","last_active":"2024-03-24T11:32:42.683810+0000","last_peered":"2024-03-24T11:32:42.683810+0000","last_clean":"2024-03-24T11:32:42.683810+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:32:42.683810+0000","last_undegraded":"2024-03-24T11:32:42.683810+0000","last_fullsized":"2024-03-24T11:32:42.683810+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:32:42.683738+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.683738+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.142199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":494,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.961811+0000","last_change":"2024-03-24T11:32:23.906538+0000","last_active":"2024-03-24T11:32:41.961811+0000","last_peered":"2024-03-24T11:32:41.961811+0000","last_clean":"2024-03-24T11:32:41.961811+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:32:41.961811+0000","last_undegraded":"2024-03-24T11:32:41.961811+0000","last_fullsized":"2024-03-24T11:32:41.961811+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:32:23.906455+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.906455+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.906085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":471,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.769407+0000","last_change":"2024-03-24T11:32:25.619271+0000","last_active":"2024-03-24T11:32:41.769407+0000","last_peered":"2024-03-24T11:32:41.769407+0000","last_clean":"2024-03-24T11:32:41.769407+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:32:41.769407+0000","last_undegraded":"2024-03-24T11:32:41.769407+0000","last_fullsized":"2024-03-24T11:32:41.769407+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:25.619184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.619184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.019241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":467,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643619+0000","last_change":"2024-03-24T11:31:35.151699+0000","last_active":"2024-03-24T11:32:41.643619+0000","last_peered":"2024-03-24T11:32:41.643619+0000","last_clean":"2024-03-24T11:32:41.643619+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:32:41.643619+0000","last_undegraded":"2024-03-24T11:32:41.643619+0000","last_fullsized":"2024-03-24T11:32:41.643619+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:35.151566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:35.151566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:44.890575+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":605,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.814852+0000","last_change":"2024-03-24T11:31:31.621753+0000","last_active":"2024-03-24T11:32:41.814852+0000","last_peered":"2024-03-24T11:32:41.814852+0000","last_clean":"2024-03-24T11:32:41.814852+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:32:41.814852+0000","last_undegraded":"2024-03-24T11:32:41.814852+0000","last_fullsized":"2024-03-24T11:32:41.814852+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.621646+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.866195+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"213'19","reported_seq":498,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643839+0000","last_change":"2024-03-24T11:32:37.784109+0000","last_active":"2024-03-24T11:32:41.643839+0000","last_peered":"2024-03-24T11:32:41.643839+0000","last_clean":"2024-03-24T11:32:41.643839+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:32:41.643839+0000","last_undegraded":"2024-03-24T11:32:41.643839+0000","last_fullsized":"2024-03-24T11:32:41.643839+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:37.784011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.784011+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:39.413955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.642907+0000","last_change":"2024-03-24T11:31:29.730215+0000","last_active":"2024-03-24T11:32:41.642907+0000","last_peered":"2024-03-24T11:32:41.642907+0000","last_clean":"2024-03-24T11:32:41.642907+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:32:41.642907+0000","last_undegraded":"2024-03-24T11:32:41.642907+0000","last_fullsized":"2024-03-24T11:32:41.642907+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.730078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.730078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:54.570128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.814564+0000","last_change":"2024-03-24T11:30:10.252564+0000","last_active":"2024-03-24T11:32:41.814564+0000","last_peered":"2024-03-24T11:32:41.814564+0000","last_clean":"2024-03-24T11:32:41.814564+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:32:41.814564+0000","last_undegraded":"2024-03-24T11:32:41.814564+0000","last_fullsized":"2024-03-24T11:32:41.814564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:10.252339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:30:10.252339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.650073+0000","last_change":"2024-03-24T11:29:39.057894+0000","last_active":"2024-03-24T11:32:41.650073+0000","last_peered":"2024-03-24T11:32:41.650073+0000","last_clean":"2024-03-24T11:32:41.650073+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:32:41.650073+0000","last_undegraded":"2024-03-24T11:32:41.650073+0000","last_fullsized":"2024-03-24T11:32:41.650073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:39.057644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:39.057644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":624,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.766063+0000","last_change":"2024-03-24T11:32:24.651691+0000","last_active":"2024-03-24T11:32:41.766063+0000","last_peered":"2024-03-24T11:32:41.766063+0000","last_clean":"2024-03-24T11:32:41.766063+0000","last_became_active":"2024-03-24T11:28:43.190009+0000","last_became_peered":"2024-03-24T11:28:43.190009+0000","last_unstale":"2024-03-24T11:32:41.766063+0000","last_undegraded":"2024-03-24T11:32:41.766063+0000","last_fullsized":"2024-03-24T11:32:41.766063+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:24.651587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.651587+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.416321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"213'34","reported_seq":525,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.768232+0000","last_change":"2024-03-24T11:32:30.766638+0000","last_active":"2024-03-24T11:32:41.768232+0000","last_peered":"2024-03-24T11:32:41.768232+0000","last_clean":"2024-03-24T11:32:41.768232+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:32:41.768232+0000","last_undegraded":"2024-03-24T11:32:41.768232+0000","last_fullsized":"2024-03-24T11:32:41.768232+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:32:30.766549+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.766549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.554399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643416+0000","last_change":"2024-03-24T11:31:22.167292+0000","last_active":"2024-03-24T11:32:41.643416+0000","last_peered":"2024-03-24T11:32:41.643416+0000","last_clean":"2024-03-24T11:32:41.643416+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:32:41.643416+0000","last_undegraded":"2024-03-24T11:32:41.643416+0000","last_fullsized":"2024-03-24T11:32:41.643416+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.167171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.167171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":629,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.644041+0000","last_change":"2024-03-24T11:32:18.783487+0000","last_active":"2024-03-24T11:32:41.644041+0000","last_peered":"2024-03-24T11:32:41.644041+0000","last_clean":"2024-03-24T11:32:41.644041+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:32:41.644041+0000","last_undegraded":"2024-03-24T11:32:41.644041+0000","last_fullsized":"2024-03-24T11:32:41.644041+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:18.783339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.783339+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:29.190654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.816357+0000","last_change":"2024-03-24T11:31:28.690070+0000","last_active":"2024-03-24T11:32:41.816357+0000","last_peered":"2024-03-24T11:32:41.816357+0000","last_clean":"2024-03-24T11:32:41.816357+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:32:41.816357+0000","last_undegraded":"2024-03-24T11:32:41.816357+0000","last_fullsized":"2024-03-24T11:32:41.816357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.689946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.689946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:55.984173+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":596,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.814960+0000","last_change":"2024-03-24T11:30:11.263503+0000","last_active":"2024-03-24T11:32:41.814960+0000","last_peered":"2024-03-24T11:32:41.814960+0000","last_clean":"2024-03-24T11:32:41.814960+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:32:41.814960+0000","last_undegraded":"2024-03-24T11:32:41.814960+0000","last_fullsized":"2024-03-24T11:32:41.814960+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:30:11.263251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.263251+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.642934+0000","last_change":"2024-03-24T11:32:39.715836+0000","last_active":"2024-03-24T11:32:41.642934+0000","last_peered":"2024-03-24T11:32:41.642934+0000","last_clean":"2024-03-24T11:32:41.642934+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:32:41.642934+0000","last_undegraded":"2024-03-24T11:32:41.642934+0000","last_fullsized":"2024-03-24T11:32:41.642934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.715759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.083140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"213'30","reported_seq":493,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643503+0000","last_change":"2024-03-24T11:31:31.099444+0000","last_active":"2024-03-24T11:32:41.643503+0000","last_peered":"2024-03-24T11:32:41.643503+0000","last_clean":"2024-03-24T11:32:41.643503+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:32:41.643503+0000","last_undegraded":"2024-03-24T11:32:41.643503+0000","last_fullsized":"2024-03-24T11:32:41.643503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:31:31.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.099209+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.955611+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.647510+0000","last_change":"2024-03-24T11:29:28.019634+0000","last_active":"2024-03-24T11:32:41.647510+0000","last_peered":"2024-03-24T11:32:41.647510+0000","last_clean":"2024-03-24T11:32:41.647510+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:32:41.647510+0000","last_undegraded":"2024-03-24T11:32:41.647510+0000","last_fullsized":"2024-03-24T11:32:41.647510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:28.018667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:29:28.018667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":443,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643049+0000","last_change":"2024-03-24T11:30:17.941383+0000","last_active":"2024-03-24T11:32:41.643049+0000","last_peered":"2024-03-24T11:32:41.643049+0000","last_clean":"2024-03-24T11:32:41.643049+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:32:41.643049+0000","last_undegraded":"2024-03-24T11:32:41.643049+0000","last_fullsized":"2024-03-24T11:32:41.643049+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.941259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":609,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643594+0000","last_change":"2024-03-24T11:31:25.692917+0000","last_active":"2024-03-24T11:32:41.643594+0000","last_peered":"2024-03-24T11:32:41.643594+0000","last_clean":"2024-03-24T11:32:41.643594+0000","last_became_active":"2024-03-24T11:28:38.173086+0000","last_became_peered":"2024-03-24T11:28:38.173086+0000","last_unstale":"2024-03-24T11:32:41.643594+0000","last_undegraded":"2024-03-24T11:32:41.643594+0000","last_fullsized":"2024-03-24T11:32:41.643594+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.692790+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:28:38.972499+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.692790+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":614,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643616+0000","last_change":"2024-03-24T11:31:08.290147+0000","last_active":"2024-03-24T11:32:41.643616+0000","last_peered":"2024-03-24T11:32:41.643616+0000","last_clean":"2024-03-24T11:32:41.643616+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:32:41.643616+0000","last_undegraded":"2024-03-24T11:32:41.643616+0000","last_fullsized":"2024-03-24T11:32:41.643616+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:08.289960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.289960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"213'28","reported_seq":501,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643030+0000","last_change":"2024-03-24T11:32:34.830458+0000","last_active":"2024-03-24T11:32:41.643030+0000","last_peered":"2024-03-24T11:32:41.643030+0000","last_clean":"2024-03-24T11:32:41.643030+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:32:41.643030+0000","last_undegraded":"2024-03-24T11:32:41.643030+0000","last_fullsized":"2024-03-24T11:32:41.643030+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:32:34.830322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.830322+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.916433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":472,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031688+0000","last_change":"2024-03-24T11:32:40.035107+0000","last_active":"2024-03-24T11:32:42.031688+0000","last_peered":"2024-03-24T11:32:42.031688+0000","last_clean":"2024-03-24T11:32:42.031688+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:32:42.031688+0000","last_undegraded":"2024-03-24T11:32:42.031688+0000","last_fullsized":"2024-03-24T11:32:42.031688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:32:40.035022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.035022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:47.238292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":83,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.962623+0000","last_change":"2024-03-24T11:32:33.957761+0000","last_active":"2024-03-24T11:32:41.962623+0000","last_peered":"2024-03-24T11:32:41.962623+0000","last_clean":"2024-03-24T11:32:41.962623+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:32:41.962623+0000","last_undegraded":"2024-03-24T11:32:41.962623+0000","last_fullsized":"2024-03-24T11:32:41.962623+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:33.957632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.957632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.301310+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":601,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.649868+0000","last_change":"2024-03-24T11:31:18.144752+0000","last_active":"2024-03-24T11:32:41.649868+0000","last_peered":"2024-03-24T11:32:41.649868+0000","last_clean":"2024-03-24T11:32:41.649868+0000","last_became_active":"2024-03-24T11:28:38.174888+0000","last_became_peered":"2024-03-24T11:28:38.174888+0000","last_unstale":"2024-03-24T11:32:41.649868+0000","last_undegraded":"2024-03-24T11:32:41.649868+0000","last_fullsized":"2024-03-24T11:32:41.649868+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.144322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.144322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":639,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.033121+0000","last_change":"2024-03-24T11:32:42.033121+0000","last_active":"2024-03-24T11:32:42.033121+0000","last_peered":"2024-03-24T11:32:42.033121+0000","last_clean":"2024-03-24T11:32:42.033121+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:32:42.033121+0000","last_undegraded":"2024-03-24T11:32:42.033121+0000","last_fullsized":"2024-03-24T11:32:42.033121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.033019+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.550557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"213'20","reported_seq":505,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.765645+0000","last_change":"2024-03-24T11:32:21.588381+0000","last_active":"2024-03-24T11:32:41.765645+0000","last_peered":"2024-03-24T11:32:41.765645+0000","last_clean":"2024-03-24T11:32:41.765645+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:32:41.765645+0000","last_undegraded":"2024-03-24T11:32:41.765645+0000","last_fullsized":"2024-03-24T11:32:41.765645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:21.588239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.588239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.355301+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":474,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.816466+0000","last_change":"2024-03-24T11:31:06.261980+0000","last_active":"2024-03-24T11:32:41.816466+0000","last_peered":"2024-03-24T11:32:41.816466+0000","last_clean":"2024-03-24T11:32:41.816466+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:32:41.816466+0000","last_undegraded":"2024-03-24T11:32:41.816466+0000","last_fullsized":"2024-03-24T11:32:41.816466+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:06.261883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.261883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":452,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.030888+0000","last_change":"2024-03-24T11:32:22.994060+0000","last_active":"2024-03-24T11:32:42.030888+0000","last_peered":"2024-03-24T11:32:42.030888+0000","last_clean":"2024-03-24T11:32:42.030888+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:32:42.030888+0000","last_undegraded":"2024-03-24T11:32:42.030888+0000","last_fullsized":"2024-03-24T11:32:42.030888+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:32:22.993851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:22.993851+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.104574+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":623,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.963140+0000","last_change":"2024-03-24T11:31:42.062272+0000","last_active":"2024-03-24T11:32:41.963140+0000","last_peered":"2024-03-24T11:32:41.963140+0000","last_clean":"2024-03-24T11:32:41.963140+0000","last_became_active":"2024-03-24T11:28:38.178260+0000","last_became_peered":"2024-03-24T11:28:38.178260+0000","last_unstale":"2024-03-24T11:32:41.963140+0000","last_undegraded":"2024-03-24T11:32:41.963140+0000","last_fullsized":"2024-03-24T11:32:41.963140+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.062151+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:10.775672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":479,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.962142+0000","last_change":"2024-03-24T11:31:33.075345+0000","last_active":"2024-03-24T11:32:41.962142+0000","last_peered":"2024-03-24T11:32:41.962142+0000","last_clean":"2024-03-24T11:32:41.962142+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:32:41.962142+0000","last_undegraded":"2024-03-24T11:32:41.962142+0000","last_fullsized":"2024-03-24T11:32:41.962142+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:33.075228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.075228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:56.584947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"213'29","reported_seq":510,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031505+0000","last_change":"2024-03-24T11:32:26.907881+0000","last_active":"2024-03-24T11:32:42.031505+0000","last_peered":"2024-03-24T11:32:42.031505+0000","last_clean":"2024-03-24T11:32:42.031505+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:32:42.031505+0000","last_undegraded":"2024-03-24T11:32:42.031505+0000","last_fullsized":"2024-03-24T11:32:42.031505+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:32:26.907769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.907769+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:32.644339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":479,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.769179+0000","last_change":"2024-03-24T11:31:25.294292+0000","last_active":"2024-03-24T11:32:41.769179+0000","last_peered":"2024-03-24T11:32:41.769179+0000","last_clean":"2024-03-24T11:32:41.769179+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:32:41.769179+0000","last_undegraded":"2024-03-24T11:32:41.769179+0000","last_fullsized":"2024-03-24T11:32:41.769179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:25.294054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.294054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:44.339730+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":459,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.764649+0000","last_change":"2024-03-24T11:31:20.405449+0000","last_active":"2024-03-24T11:32:41.764649+0000","last_peered":"2024-03-24T11:32:41.764649+0000","last_clean":"2024-03-24T11:32:41.764649+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:32:41.764649+0000","last_undegraded":"2024-03-24T11:32:41.764649+0000","last_fullsized":"2024-03-24T11:32:41.764649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:20.405340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:20.405340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:42.309256+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":623,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.962307+0000","last_change":"2024-03-24T11:31:18.894161+0000","last_active":"2024-03-24T11:32:41.962307+0000","last_peered":"2024-03-24T11:32:41.962307+0000","last_clean":"2024-03-24T11:32:41.962307+0000","last_became_active":"2024-03-24T11:28:38.174742+0000","last_became_peered":"2024-03-24T11:28:38.174742+0000","last_unstale":"2024-03-24T11:32:41.962307+0000","last_undegraded":"2024-03-24T11:32:41.962307+0000","last_fullsized":"2024-03-24T11:32:41.962307+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.894050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.894050+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.822017+0000","last_change":"2024-03-24T11:30:27.391730+0000","last_active":"2024-03-24T11:32:41.822017+0000","last_peered":"2024-03-24T11:32:41.822017+0000","last_clean":"2024-03-24T11:32:41.822017+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:32:41.822017+0000","last_undegraded":"2024-03-24T11:32:41.822017+0000","last_fullsized":"2024-03-24T11:32:41.822017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:30:27.391607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"213'24","reported_seq":495,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031582+0000","last_change":"2024-03-24T11:32:21.972798+0000","last_active":"2024-03-24T11:32:42.031582+0000","last_peered":"2024-03-24T11:32:42.031582+0000","last_clean":"2024-03-24T11:32:42.031582+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:32:42.031582+0000","last_undegraded":"2024-03-24T11:32:42.031582+0000","last_fullsized":"2024-03-24T11:32:42.031582+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:21.972644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.972644+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.486043+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":255,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.869223+0000","last_change":"2024-03-24T11:32:39.822026+0000","last_active":"2024-03-24T11:32:42.869223+0000","last_peered":"2024-03-24T11:32:42.869223+0000","last_clean":"2024-03-24T11:32:42.869223+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:32:42.869223+0000","last_undegraded":"2024-03-24T11:32:42.869223+0000","last_fullsized":"2024-03-24T11:32:42.869223+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:32:39.821933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.821933+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.501549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":130,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":466,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031392+0000","last_change":"2024-03-24T11:31:21.629091+0000","last_active":"2024-03-24T11:32:42.031392+0000","last_peered":"2024-03-24T11:32:42.031392+0000","last_clean":"2024-03-24T11:32:42.031392+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:32:42.031392+0000","last_undegraded":"2024-03-24T11:32:42.031392+0000","last_fullsized":"2024-03-24T11:32:42.031392+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:21.628978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:21.628978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":467,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.768979+0000","last_change":"2024-03-24T11:31:42.720752+0000","last_active":"2024-03-24T11:32:41.768979+0000","last_peered":"2024-03-24T11:32:41.768979+0000","last_clean":"2024-03-24T11:32:41.768979+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:32:41.768979+0000","last_undegraded":"2024-03-24T11:32:41.768979+0000","last_fullsized":"2024-03-24T11:32:41.768979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:42.720504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.720504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:00.102566+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":617,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.766164+0000","last_change":"2024-03-24T11:32:26.629982+0000","last_active":"2024-03-24T11:32:41.766164+0000","last_peered":"2024-03-24T11:32:41.766164+0000","last_clean":"2024-03-24T11:32:41.766164+0000","last_became_active":"2024-03-24T11:28:38.173843+0000","last_became_peered":"2024-03-24T11:28:38.173843+0000","last_unstale":"2024-03-24T11:32:41.766164+0000","last_undegraded":"2024-03-24T11:32:41.766164+0000","last_fullsized":"2024-03-24T11:32:41.766164+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:26.629894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.629894+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.767066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":83,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.644364+0000","last_change":"2024-03-24T11:32:41.207872+0000","last_active":"2024-03-24T11:32:41.644364+0000","last_peered":"2024-03-24T11:32:41.644364+0000","last_clean":"2024-03-24T11:32:41.644364+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:32:41.644364+0000","last_undegraded":"2024-03-24T11:32:41.644364+0000","last_fullsized":"2024-03-24T11:32:41.644364+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:41.207792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.207792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.669676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"213'36","reported_seq":697,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.962871+0000","last_change":"2024-03-24T11:32:25.853007+0000","last_active":"2024-03-24T11:32:41.962871+0000","last_peered":"2024-03-24T11:32:41.962871+0000","last_clean":"2024-03-24T11:32:41.962871+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:32:41.962871+0000","last_undegraded":"2024-03-24T11:32:41.962871+0000","last_fullsized":"2024-03-24T11:32:41.962871+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'36","last_scrub_stamp":"2024-03-24T11:32:25.852913+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.852913+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.114595+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":461,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646099+0000","last_change":"2024-03-24T11:32:28.089047+0000","last_active":"2024-03-24T11:32:41.646099+0000","last_peered":"2024-03-24T11:32:41.646099+0000","last_clean":"2024-03-24T11:32:41.646099+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:32:41.646099+0000","last_undegraded":"2024-03-24T11:32:41.646099+0000","last_fullsized":"2024-03-24T11:32:41.646099+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.088969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.088969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.266746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":371,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031767+0000","last_change":"2024-03-24T11:31:07.885631+0000","last_active":"2024-03-24T11:32:42.031767+0000","last_peered":"2024-03-24T11:32:42.031767+0000","last_clean":"2024-03-24T11:32:42.031767+0000","last_became_active":"2024-03-24T11:30:35.355009+0000","last_became_peered":"2024-03-24T11:30:35.355009+0000","last_unstale":"2024-03-24T11:32:42.031767+0000","last_undegraded":"2024-03-24T11:32:42.031767+0000","last_fullsized":"2024-03-24T11:32:42.031767+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:07.885508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:07.885508+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":467,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.821733+0000","last_change":"2024-03-24T11:31:26.676287+0000","last_active":"2024-03-24T11:32:41.821733+0000","last_peered":"2024-03-24T11:32:41.821733+0000","last_clean":"2024-03-24T11:32:41.821733+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:32:41.821733+0000","last_undegraded":"2024-03-24T11:32:41.821733+0000","last_fullsized":"2024-03-24T11:32:41.821733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:26.676209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:31:26.676209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":471,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.644491+0000","last_change":"2024-03-24T11:31:31.994930+0000","last_active":"2024-03-24T11:32:41.644491+0000","last_peered":"2024-03-24T11:32:41.644491+0000","last_clean":"2024-03-24T11:32:41.644491+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:32:41.644491+0000","last_undegraded":"2024-03-24T11:32:41.644491+0000","last_fullsized":"2024-03-24T11:32:41.644491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:31.994604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.994604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:42.774652+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"213'17","reported_seq":483,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643389+0000","last_change":"2024-03-24T11:32:38.137085+0000","last_active":"2024-03-24T11:32:41.643389+0000","last_peered":"2024-03-24T11:32:41.643389+0000","last_clean":"2024-03-24T11:32:41.643389+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:32:41.643389+0000","last_undegraded":"2024-03-24T11:32:41.643389+0000","last_fullsized":"2024-03-24T11:32:41.643389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'17","last_scrub_stamp":"2024-03-24T11:32:38.136966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.136966+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.504065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":488,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.642999+0000","last_change":"2024-03-24T11:32:18.855130+0000","last_active":"2024-03-24T11:32:41.642999+0000","last_peered":"2024-03-24T11:32:41.642999+0000","last_clean":"2024-03-24T11:32:41.642999+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:32:41.642999+0000","last_undegraded":"2024-03-24T11:32:41.642999+0000","last_fullsized":"2024-03-24T11:32:41.642999+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:18.854979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.854979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:20.771742+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":479,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.765101+0000","last_change":"2024-03-24T11:31:29.443192+0000","last_active":"2024-03-24T11:32:41.765101+0000","last_peered":"2024-03-24T11:32:41.765101+0000","last_clean":"2024-03-24T11:32:41.765101+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:32:41.765101+0000","last_undegraded":"2024-03-24T11:32:41.765101+0000","last_fullsized":"2024-03-24T11:32:41.765101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.443013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.443013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.737539+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":83,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.994801+0000","last_change":"2024-03-24T11:32:42.994801+0000","last_active":"2024-03-24T11:32:42.994801+0000","last_peered":"2024-03-24T11:32:42.994801+0000","last_clean":"2024-03-24T11:32:42.994801+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:32:42.994801+0000","last_undegraded":"2024-03-24T11:32:42.994801+0000","last_fullsized":"2024-03-24T11:32:42.994801+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.994694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.994694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.156201+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":371,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031227+0000","last_change":"2024-03-24T11:31:08.903891+0000","last_active":"2024-03-24T11:32:42.031227+0000","last_peered":"2024-03-24T11:32:42.031227+0000","last_clean":"2024-03-24T11:32:42.031227+0000","last_became_active":"2024-03-24T11:30:35.356166+0000","last_became_peered":"2024-03-24T11:30:35.356166+0000","last_unstale":"2024-03-24T11:32:42.031227+0000","last_undegraded":"2024-03-24T11:32:42.031227+0000","last_fullsized":"2024-03-24T11:32:42.031227+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:08.903763+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.903763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.030663+0000","last_change":"2024-03-24T11:31:28.613941+0000","last_active":"2024-03-24T11:32:42.030663+0000","last_peered":"2024-03-24T11:32:42.030663+0000","last_clean":"2024-03-24T11:32:42.030663+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:32:42.030663+0000","last_undegraded":"2024-03-24T11:32:42.030663+0000","last_fullsized":"2024-03-24T11:32:42.030663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.613752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.613752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"213'40","reported_seq":708,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:43.782374+0000","last_change":"2024-03-24T11:32:43.782374+0000","last_active":"2024-03-24T11:32:43.782374+0000","last_peered":"2024-03-24T11:32:43.782374+0000","last_clean":"2024-03-24T11:32:43.782374+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:32:43.782374+0000","last_undegraded":"2024-03-24T11:32:43.782374+0000","last_fullsized":"2024-03-24T11:32:43.782374+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'40","last_scrub_stamp":"2024-03-24T11:32:43.782256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.782256+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:01.406173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643100+0000","last_change":"2024-03-24T11:31:05.775644+0000","last_active":"2024-03-24T11:32:41.643100+0000","last_peered":"2024-03-24T11:32:41.643100+0000","last_clean":"2024-03-24T11:32:41.643100+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:32:41.643100+0000","last_undegraded":"2024-03-24T11:32:41.643100+0000","last_fullsized":"2024-03-24T11:32:41.643100+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:05.775534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:05.775534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":482,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815159+0000","last_change":"2024-03-24T11:32:28.707278+0000","last_active":"2024-03-24T11:32:41.815159+0000","last_peered":"2024-03-24T11:32:41.815159+0000","last_clean":"2024-03-24T11:32:41.815159+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:32:41.815159+0000","last_undegraded":"2024-03-24T11:32:41.815159+0000","last_fullsized":"2024-03-24T11:32:41.815159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:32:28.707184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.707184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:33.691220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646546+0000","last_change":"2024-03-24T11:29:18.915411+0000","last_active":"2024-03-24T11:32:41.646546+0000","last_peered":"2024-03-24T11:32:41.646546+0000","last_clean":"2024-03-24T11:32:41.646546+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:32:41.646546+0000","last_undegraded":"2024-03-24T11:32:41.646546+0000","last_fullsized":"2024-03-24T11:32:41.646546+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:18.915289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:18.915289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":617,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815487+0000","last_change":"2024-03-24T11:32:29.691475+0000","last_active":"2024-03-24T11:32:41.815487+0000","last_peered":"2024-03-24T11:32:41.815487+0000","last_clean":"2024-03-24T11:32:41.815487+0000","last_became_active":"2024-03-24T11:28:38.173832+0000","last_became_peered":"2024-03-24T11:28:38.173832+0000","last_unstale":"2024-03-24T11:32:41.815487+0000","last_undegraded":"2024-03-24T11:32:41.815487+0000","last_fullsized":"2024-03-24T11:32:41.815487+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:29.691383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:29.691383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.888681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"213'15","reported_seq":492,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031743+0000","last_change":"2024-03-24T11:32:37.060211+0000","last_active":"2024-03-24T11:32:42.031743+0000","last_peered":"2024-03-24T11:32:42.031743+0000","last_clean":"2024-03-24T11:32:42.031743+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:32:42.031743+0000","last_undegraded":"2024-03-24T11:32:42.031743+0000","last_fullsized":"2024-03-24T11:32:42.031743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:32:37.060067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.060067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.865083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":477,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815599+0000","last_change":"2024-03-24T11:31:20.465180+0000","last_active":"2024-03-24T11:32:41.815599+0000","last_peered":"2024-03-24T11:32:41.815599+0000","last_clean":"2024-03-24T11:32:41.815599+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:32:41.815599+0000","last_undegraded":"2024-03-24T11:32:41.815599+0000","last_fullsized":"2024-03-24T11:32:41.815599+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:20.465096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:20.465096+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.122475+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":471,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643814+0000","last_change":"2024-03-24T11:32:27.943222+0000","last_active":"2024-03-24T11:32:41.643814+0000","last_peered":"2024-03-24T11:32:41.643814+0000","last_clean":"2024-03-24T11:32:41.643814+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:32:41.643814+0000","last_undegraded":"2024-03-24T11:32:41.643814+0000","last_fullsized":"2024-03-24T11:32:41.643814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.943114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.943114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.987411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031656+0000","last_change":"2024-03-24T11:31:23.762948+0000","last_active":"2024-03-24T11:32:42.031656+0000","last_peered":"2024-03-24T11:32:42.031656+0000","last_clean":"2024-03-24T11:32:42.031656+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:32:42.031656+0000","last_undegraded":"2024-03-24T11:32:42.031656+0000","last_fullsized":"2024-03-24T11:32:42.031656+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:23.762486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:23.762486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:43.614566+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":448,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643639+0000","last_change":"2024-03-24T11:31:33.687541+0000","last_active":"2024-03-24T11:32:41.643639+0000","last_peered":"2024-03-24T11:32:41.643639+0000","last_clean":"2024-03-24T11:32:41.643639+0000","last_became_active":"2024-03-24T11:28:43.192246+0000","last_became_peered":"2024-03-24T11:28:43.192246+0000","last_unstale":"2024-03-24T11:32:41.643639+0000","last_undegraded":"2024-03-24T11:32:41.643639+0000","last_fullsized":"2024-03-24T11:32:41.643639+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:33.687261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.687261+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:43.601587+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"213'44","reported_seq":539,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.962037+0000","last_change":"2024-03-24T11:31:21.903188+0000","last_active":"2024-03-24T11:32:41.962037+0000","last_peered":"2024-03-24T11:32:41.962037+0000","last_clean":"2024-03-24T11:32:41.962037+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:32:41.962037+0000","last_undegraded":"2024-03-24T11:32:41.962037+0000","last_fullsized":"2024-03-24T11:32:41.962037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:31:21.903023+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:31:21.903023+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":74,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.816570+0000","last_change":"2024-03-24T11:32:38.827248+0000","last_active":"2024-03-24T11:32:41.816570+0000","last_peered":"2024-03-24T11:32:41.816570+0000","last_clean":"2024-03-24T11:32:41.816570+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:32:41.816570+0000","last_undegraded":"2024-03-24T11:32:41.816570+0000","last_fullsized":"2024-03-24T11:32:41.816570+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.827167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.827167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.576619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":471,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643658+0000","last_change":"2024-03-24T11:31:22.849153+0000","last_active":"2024-03-24T11:32:41.643658+0000","last_peered":"2024-03-24T11:32:41.643658+0000","last_clean":"2024-03-24T11:32:41.643658+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:32:41.643658+0000","last_undegraded":"2024-03-24T11:32:41.643658+0000","last_fullsized":"2024-03-24T11:32:41.643658+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.849049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.849049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:48.408647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":74,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643017+0000","last_change":"2024-03-24T11:31:32.996560+0000","last_active":"2024-03-24T11:32:41.643017+0000","last_peered":"2024-03-24T11:32:41.643017+0000","last_clean":"2024-03-24T11:32:41.643017+0000","last_became_active":"2024-03-24T11:30:35.355210+0000","last_became_peered":"2024-03-24T11:30:35.355210+0000","last_unstale":"2024-03-24T11:32:41.643017+0000","last_undegraded":"2024-03-24T11:32:41.643017+0000","last_fullsized":"2024-03-24T11:32:41.643017+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:32.996416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:32.996416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:41.925710+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":630,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.963077+0000","last_change":"2024-03-24T11:32:34.973136+0000","last_active":"2024-03-24T11:32:41.963077+0000","last_peered":"2024-03-24T11:32:41.963077+0000","last_clean":"2024-03-24T11:32:41.963077+0000","last_became_active":"2024-03-24T11:28:43.191874+0000","last_became_peered":"2024-03-24T11:28:43.191874+0000","last_unstale":"2024-03-24T11:32:41.963077+0000","last_undegraded":"2024-03-24T11:32:41.963077+0000","last_fullsized":"2024-03-24T11:32:41.963077+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.973008+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.990059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":471,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.642781+0000","last_change":"2024-03-24T11:31:36.047503+0000","last_active":"2024-03-24T11:32:41.642781+0000","last_peered":"2024-03-24T11:32:41.642781+0000","last_clean":"2024-03-24T11:32:41.642781+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:32:41.642781+0000","last_undegraded":"2024-03-24T11:32:41.642781+0000","last_fullsized":"2024-03-24T11:32:41.642781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:36.047102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.047102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"213'18","reported_seq":491,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643190+0000","last_change":"2024-03-24T11:31:11.798229+0000","last_active":"2024-03-24T11:32:41.643190+0000","last_peered":"2024-03-24T11:32:41.643190+0000","last_clean":"2024-03-24T11:32:41.643190+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:32:41.643190+0000","last_undegraded":"2024-03-24T11:32:41.643190+0000","last_fullsized":"2024-03-24T11:32:41.643190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.798089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.798089+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":66,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643791+0000","last_change":"2024-03-24T11:30:35.363477+0000","last_active":"2024-03-24T11:32:41.643791+0000","last_peered":"2024-03-24T11:32:41.643791+0000","last_clean":"2024-03-24T11:32:41.643791+0000","last_became_active":"2024-03-24T11:30:35.356055+0000","last_became_peered":"2024-03-24T11:30:35.356055+0000","last_unstale":"2024-03-24T11:32:41.643791+0000","last_undegraded":"2024-03-24T11:32:41.643791+0000","last_fullsized":"2024-03-24T11:32:41.643791+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:07.709842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:30:07.709842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.649020+0000","last_change":"2024-03-24T11:30:12.206577+0000","last_active":"2024-03-24T11:32:41.649020+0000","last_peered":"2024-03-24T11:32:41.649020+0000","last_clean":"2024-03-24T11:32:41.649020+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:32:41.649020+0000","last_undegraded":"2024-03-24T11:32:41.649020+0000","last_fullsized":"2024-03-24T11:32:41.649020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:12.206287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:12.206287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":606,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646204+0000","last_change":"2024-03-24T11:29:10.818100+0000","last_active":"2024-03-24T11:32:41.646204+0000","last_peered":"2024-03-24T11:32:41.646204+0000","last_clean":"2024-03-24T11:32:41.646204+0000","last_became_active":"2024-03-24T11:28:43.189494+0000","last_became_peered":"2024-03-24T11:28:43.189494+0000","last_unstale":"2024-03-24T11:32:41.646204+0000","last_undegraded":"2024-03-24T11:32:41.646204+0000","last_fullsized":"2024-03-24T11:32:41.646204+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:29:10.817950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:29:10.817950+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"238'9","reported_seq":125,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.962145+0000","last_change":"2024-03-24T11:31:30.056940+0000","last_active":"2024-03-24T11:32:41.962145+0000","last_peered":"2024-03-24T11:32:41.962145+0000","last_clean":"2024-03-24T11:32:41.962145+0000","last_became_active":"2024-03-24T11:30:35.359635+0000","last_became_peered":"2024-03-24T11:30:35.359635+0000","last_unstale":"2024-03-24T11:32:41.962145+0000","last_undegraded":"2024-03-24T11:32:41.962145+0000","last_fullsized":"2024-03-24T11:32:41.962145+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'9","last_scrub_stamp":"2024-03-24T11:31:30.056809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.056809+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"213'34","reported_seq":517,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.814604+0000","last_change":"2024-03-24T11:31:12.435804+0000","last_active":"2024-03-24T11:32:41.814604+0000","last_peered":"2024-03-24T11:32:41.814604+0000","last_clean":"2024-03-24T11:32:41.814604+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:32:41.814604+0000","last_undegraded":"2024-03-24T11:32:41.814604+0000","last_fullsized":"2024-03-24T11:32:41.814604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:31:12.435639+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:31:12.435639+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":268,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643884+0000","last_change":"2024-03-24T11:30:35.356491+0000","last_active":"2024-03-24T11:32:41.643884+0000","last_peered":"2024-03-24T11:32:41.643884+0000","last_clean":"2024-03-24T11:32:41.643884+0000","last_became_active":"2024-03-24T11:30:35.356100+0000","last_became_peered":"2024-03-24T11:30:35.356100+0000","last_unstale":"2024-03-24T11:32:41.643884+0000","last_undegraded":"2024-03-24T11:32:41.643884+0000","last_fullsized":"2024-03-24T11:32:41.643884+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:30:11.625473+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.625473+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":459,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815401+0000","last_change":"2024-03-24T11:31:14.458586+0000","last_active":"2024-03-24T11:32:41.815401+0000","last_peered":"2024-03-24T11:32:41.815401+0000","last_clean":"2024-03-24T11:32:41.815401+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:32:41.815401+0000","last_undegraded":"2024-03-24T11:32:41.815401+0000","last_fullsized":"2024-03-24T11:32:41.815401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.458470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.458470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":533,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643566+0000","last_change":"2024-03-24T11:31:04.735517+0000","last_active":"2024-03-24T11:32:41.643566+0000","last_peered":"2024-03-24T11:32:41.643566+0000","last_clean":"2024-03-24T11:32:41.643566+0000","last_became_active":"2024-03-24T11:30:35.354637+0000","last_became_peered":"2024-03-24T11:30:35.354637+0000","last_unstale":"2024-03-24T11:32:41.643566+0000","last_undegraded":"2024-03-24T11:32:41.643566+0000","last_fullsized":"2024-03-24T11:32:41.643566+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:04.735433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:04.735433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"213'15","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.647787+0000","last_change":"2024-03-24T11:30:14.150582+0000","last_active":"2024-03-24T11:32:41.647787+0000","last_peered":"2024-03-24T11:32:41.647787+0000","last_clean":"2024-03-24T11:32:41.647787+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:32:41.647787+0000","last_undegraded":"2024-03-24T11:32:41.647787+0000","last_fullsized":"2024-03-24T11:32:41.647787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:14.150454+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.150454+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":486,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.961784+0000","last_change":"2024-03-24T11:32:40.958471+0000","last_active":"2024-03-24T11:32:41.961784+0000","last_peered":"2024-03-24T11:32:41.961784+0000","last_clean":"2024-03-24T11:32:41.961784+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:32:41.961784+0000","last_undegraded":"2024-03-24T11:32:41.961784+0000","last_fullsized":"2024-03-24T11:32:41.961784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:40.958389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.958389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.190213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.814522+0000","last_change":"2024-03-24T11:32:39.835187+0000","last_active":"2024-03-24T11:32:41.814522+0000","last_peered":"2024-03-24T11:32:41.814522+0000","last_clean":"2024-03-24T11:32:41.814522+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:32:41.814522+0000","last_undegraded":"2024-03-24T11:32:41.814522+0000","last_fullsized":"2024-03-24T11:32:41.814522+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.835097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.835097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:03.202024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":82,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.963070+0000","last_change":"2024-03-24T11:32:32.920081+0000","last_active":"2024-03-24T11:32:41.963070+0000","last_peered":"2024-03-24T11:32:41.963070+0000","last_clean":"2024-03-24T11:32:41.963070+0000","last_became_active":"2024-03-24T11:30:35.353212+0000","last_became_peered":"2024-03-24T11:30:35.353212+0000","last_unstale":"2024-03-24T11:32:41.963070+0000","last_undegraded":"2024-03-24T11:32:41.963070+0000","last_fullsized":"2024-03-24T11:32:41.963070+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.919963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:34.622437+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":448,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.814853+0000","last_change":"2024-03-24T11:31:24.533967+0000","last_active":"2024-03-24T11:32:41.814853+0000","last_peered":"2024-03-24T11:32:41.814853+0000","last_clean":"2024-03-24T11:32:41.814853+0000","last_became_active":"2024-03-24T11:28:43.189410+0000","last_became_peered":"2024-03-24T11:28:43.189410+0000","last_unstale":"2024-03-24T11:32:41.814853+0000","last_undegraded":"2024-03-24T11:32:41.814853+0000","last_fullsized":"2024-03-24T11:32:41.814853+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_clean_scrub_stamp":"2024-03-24T11:31:24.533874+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:51.862202+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":624,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.765482+0000","last_change":"2024-03-24T11:32:33.790867+0000","last_active":"2024-03-24T11:32:41.765482+0000","last_peered":"2024-03-24T11:32:41.765482+0000","last_clean":"2024-03-24T11:32:41.765482+0000","last_became_active":"2024-03-24T11:28:43.192449+0000","last_became_peered":"2024-03-24T11:28:43.192449+0000","last_unstale":"2024-03-24T11:32:41.765482+0000","last_undegraded":"2024-03-24T11:32:41.765482+0000","last_fullsized":"2024-03-24T11:32:41.765482+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.790777+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.025287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"213'19","reported_seq":495,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031918+0000","last_change":"2024-03-24T11:32:32.023601+0000","last_active":"2024-03-24T11:32:42.031918+0000","last_peered":"2024-03-24T11:32:42.031918+0000","last_clean":"2024-03-24T11:32:42.031918+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:32:42.031918+0000","last_undegraded":"2024-03-24T11:32:42.031918+0000","last_fullsized":"2024-03-24T11:32:42.031918+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:32.023502+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.023502+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.100674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":66,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643074+0000","last_change":"2024-03-24T11:30:35.359322+0000","last_active":"2024-03-24T11:32:41.643074+0000","last_peered":"2024-03-24T11:32:41.643074+0000","last_clean":"2024-03-24T11:32:41.643074+0000","last_became_active":"2024-03-24T11:30:35.358928+0000","last_became_peered":"2024-03-24T11:30:35.358928+0000","last_unstale":"2024-03-24T11:32:41.643074+0000","last_undegraded":"2024-03-24T11:32:41.643074+0000","last_fullsized":"2024-03-24T11:32:41.643074+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:21.749544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:21.749544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.647706+0000","last_change":"2024-03-24T11:29:26.014447+0000","last_active":"2024-03-24T11:32:41.647706+0000","last_peered":"2024-03-24T11:32:41.647706+0000","last_clean":"2024-03-24T11:32:41.647706+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:32:41.647706+0000","last_undegraded":"2024-03-24T11:32:41.647706+0000","last_fullsized":"2024-03-24T11:32:41.647706+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:26.014199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:29:26.014199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":466,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.644231+0000","last_change":"2024-03-24T11:32:26.898659+0000","last_active":"2024-03-24T11:32:41.644231+0000","last_peered":"2024-03-24T11:32:41.644231+0000","last_clean":"2024-03-24T11:32:41.644231+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:32:41.644231+0000","last_undegraded":"2024-03-24T11:32:41.644231+0000","last_fullsized":"2024-03-24T11:32:41.644231+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:26.898583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.898583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.612791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"213'15","reported_seq":491,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.816580+0000","last_change":"2024-03-24T11:31:27.541622+0000","last_active":"2024-03-24T11:32:41.816580+0000","last_peered":"2024-03-24T11:32:41.816580+0000","last_clean":"2024-03-24T11:32:41.816580+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:32:41.816580+0000","last_undegraded":"2024-03-24T11:32:41.816580+0000","last_fullsized":"2024-03-24T11:32:41.816580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:27.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.541510+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:52.331987+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031102+0000","last_change":"2024-03-24T11:30:26.479018+0000","last_active":"2024-03-24T11:32:42.031102+0000","last_peered":"2024-03-24T11:32:42.031102+0000","last_clean":"2024-03-24T11:32:42.031102+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:32:42.031102+0000","last_undegraded":"2024-03-24T11:32:42.031102+0000","last_fullsized":"2024-03-24T11:32:42.031102+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:26.478900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:30:26.478900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":621,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.650530+0000","last_change":"2024-03-24T11:31:13.201069+0000","last_active":"2024-03-24T11:32:41.650530+0000","last_peered":"2024-03-24T11:32:41.650530+0000","last_clean":"2024-03-24T11:32:41.650530+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:32:41.650530+0000","last_undegraded":"2024-03-24T11:32:41.650530+0000","last_fullsized":"2024-03-24T11:32:41.650530+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:13.200503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.200503+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643943+0000","last_change":"2024-03-24T11:32:27.764304+0000","last_active":"2024-03-24T11:32:41.643943+0000","last_peered":"2024-03-24T11:32:41.643943+0000","last_clean":"2024-03-24T11:32:41.643943+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:32:41.643943+0000","last_undegraded":"2024-03-24T11:32:41.643943+0000","last_fullsized":"2024-03-24T11:32:41.643943+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.764203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.764203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.837847+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":448,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.814910+0000","last_change":"2024-03-24T11:31:25.526652+0000","last_active":"2024-03-24T11:32:41.814910+0000","last_peered":"2024-03-24T11:32:41.814910+0000","last_clean":"2024-03-24T11:32:41.814910+0000","last_became_active":"2024-03-24T11:28:43.188941+0000","last_became_peered":"2024-03-24T11:28:43.188941+0000","last_unstale":"2024-03-24T11:32:41.814910+0000","last_undegraded":"2024-03-24T11:32:41.814910+0000","last_fullsized":"2024-03-24T11:32:41.814910+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.526522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"213'20","reported_seq":472,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.650330+0000","last_change":"2024-03-24T11:29:32.055362+0000","last_active":"2024-03-24T11:32:41.650330+0000","last_peered":"2024-03-24T11:32:41.650330+0000","last_clean":"2024-03-24T11:32:41.650330+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:32:41.650330+0000","last_undegraded":"2024-03-24T11:32:41.650330+0000","last_fullsized":"2024-03-24T11:32:41.650330+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:29:32.055233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:29:32.055233+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":74,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643330+0000","last_change":"2024-03-24T11:31:30.728130+0000","last_active":"2024-03-24T11:32:41.643330+0000","last_peered":"2024-03-24T11:32:41.643330+0000","last_clean":"2024-03-24T11:32:41.643330+0000","last_became_active":"2024-03-24T11:30:35.353946+0000","last_became_peered":"2024-03-24T11:30:35.353946+0000","last_unstale":"2024-03-24T11:32:41.643330+0000","last_undegraded":"2024-03-24T11:32:41.643330+0000","last_fullsized":"2024-03-24T11:32:41.643330+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:30.728004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.728004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:49.089471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":636,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.816466+0000","last_change":"2024-03-24T11:31:36.682644+0000","last_active":"2024-03-24T11:32:41.816466+0000","last_peered":"2024-03-24T11:32:41.816466+0000","last_clean":"2024-03-24T11:32:41.816466+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:32:41.816466+0000","last_undegraded":"2024-03-24T11:32:41.816466+0000","last_fullsized":"2024-03-24T11:32:41.816466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:36.682542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.682542+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.160138+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031481+0000","last_change":"2024-03-24T11:32:31.071719+0000","last_active":"2024-03-24T11:32:42.031481+0000","last_peered":"2024-03-24T11:32:42.031481+0000","last_clean":"2024-03-24T11:32:42.031481+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:32:42.031481+0000","last_undegraded":"2024-03-24T11:32:42.031481+0000","last_fullsized":"2024-03-24T11:32:42.031481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:31.071621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:31.071621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.164745+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031083+0000","last_change":"2024-03-24T11:31:06.835419+0000","last_active":"2024-03-24T11:32:42.031083+0000","last_peered":"2024-03-24T11:32:42.031083+0000","last_clean":"2024-03-24T11:32:42.031083+0000","last_became_active":"2024-03-24T11:28:43.188012+0000","last_became_peered":"2024-03-24T11:28:43.188012+0000","last_unstale":"2024-03-24T11:32:42.031083+0000","last_undegraded":"2024-03-24T11:32:42.031083+0000","last_fullsized":"2024-03-24T11:32:42.031083+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:06.835322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.835322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"213'24","reported_seq":494,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643418+0000","last_change":"2024-03-24T11:30:13.928893+0000","last_active":"2024-03-24T11:32:41.643418+0000","last_peered":"2024-03-24T11:32:41.643418+0000","last_clean":"2024-03-24T11:32:41.643418+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:32:41.643418+0000","last_undegraded":"2024-03-24T11:32:41.643418+0000","last_fullsized":"2024-03-24T11:32:41.643418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:30:13.928734+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:30:13.928734+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643696+0000","last_change":"2024-03-24T11:31:37.636078+0000","last_active":"2024-03-24T11:32:41.643696+0000","last_peered":"2024-03-24T11:32:41.643696+0000","last_clean":"2024-03-24T11:32:41.643696+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:32:41.643696+0000","last_undegraded":"2024-03-24T11:32:41.643696+0000","last_fullsized":"2024-03-24T11:32:41.643696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:37.635705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.635705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:46.929525+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":478,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.765580+0000","last_change":"2024-03-24T11:32:36.752484+0000","last_active":"2024-03-24T11:32:41.765580+0000","last_peered":"2024-03-24T11:32:41.765580+0000","last_clean":"2024-03-24T11:32:41.765580+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:32:41.765580+0000","last_undegraded":"2024-03-24T11:32:41.765580+0000","last_fullsized":"2024-03-24T11:32:41.765580+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.752406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.752406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:47.500544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":442,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.648433+0000","last_change":"2024-03-24T11:32:32.077273+0000","last_active":"2024-03-24T11:32:41.648433+0000","last_peered":"2024-03-24T11:32:41.648433+0000","last_clean":"2024-03-24T11:32:41.648433+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:32:41.648433+0000","last_undegraded":"2024-03-24T11:32:41.648433+0000","last_fullsized":"2024-03-24T11:32:41.648433+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.077150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.077150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:35.573524+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"213'15","reported_seq":491,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.765698+0000","last_change":"2024-03-24T11:31:36.651472+0000","last_active":"2024-03-24T11:32:41.765698+0000","last_peered":"2024-03-24T11:32:41.765698+0000","last_clean":"2024-03-24T11:32:41.765698+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:32:41.765698+0000","last_undegraded":"2024-03-24T11:32:41.765698+0000","last_fullsized":"2024-03-24T11:32:41.765698+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:36.651340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.651340+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:49.260143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":66,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646875+0000","last_change":"2024-03-24T11:30:35.353582+0000","last_active":"2024-03-24T11:32:41.646875+0000","last_peered":"2024-03-24T11:32:41.646875+0000","last_clean":"2024-03-24T11:32:41.646875+0000","last_became_active":"2024-03-24T11:30:35.353230+0000","last_became_peered":"2024-03-24T11:30:35.353230+0000","last_unstale":"2024-03-24T11:32:41.646875+0000","last_undegraded":"2024-03-24T11:32:41.646875+0000","last_fullsized":"2024-03-24T11:32:41.646875+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:23.813152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:29:23.813152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646573+0000","last_change":"2024-03-24T11:29:29.048318+0000","last_active":"2024-03-24T11:32:41.646573+0000","last_peered":"2024-03-24T11:32:41.646573+0000","last_clean":"2024-03-24T11:32:41.646573+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:32:41.646573+0000","last_undegraded":"2024-03-24T11:32:41.646573+0000","last_fullsized":"2024-03-24T11:32:41.646573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:29.046315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:29:29.046315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031121+0000","last_change":"2024-03-24T11:32:24.051941+0000","last_active":"2024-03-24T11:32:42.031121+0000","last_peered":"2024-03-24T11:32:42.031121+0000","last_clean":"2024-03-24T11:32:42.031121+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:32:42.031121+0000","last_undegraded":"2024-03-24T11:32:42.031121+0000","last_fullsized":"2024-03-24T11:32:42.031121+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.051777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.051777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.050306+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":611,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643756+0000","last_change":"2024-03-24T11:32:40.219356+0000","last_active":"2024-03-24T11:32:41.643756+0000","last_peered":"2024-03-24T11:32:41.643756+0000","last_clean":"2024-03-24T11:32:41.643756+0000","last_became_active":"2024-03-24T11:28:43.192919+0000","last_became_peered":"2024-03-24T11:28:43.192919+0000","last_unstale":"2024-03-24T11:32:41.643756+0000","last_undegraded":"2024-03-24T11:32:41.643756+0000","last_fullsized":"2024-03-24T11:32:41.643756+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:40.219272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.219272+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.153521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"213'19","reported_seq":473,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.648708+0000","last_change":"2024-03-24T11:30:13.139116+0000","last_active":"2024-03-24T11:32:41.648708+0000","last_peered":"2024-03-24T11:32:41.648708+0000","last_clean":"2024-03-24T11:32:41.648708+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:32:41.648708+0000","last_undegraded":"2024-03-24T11:32:41.648708+0000","last_fullsized":"2024-03-24T11:32:41.648708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:30:13.138980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:13.138980+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":470,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.816583+0000","last_change":"2024-03-24T11:31:40.668063+0000","last_active":"2024-03-24T11:32:41.816583+0000","last_peered":"2024-03-24T11:32:41.816583+0000","last_clean":"2024-03-24T11:32:41.816583+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:32:41.816583+0000","last_undegraded":"2024-03-24T11:32:41.816583+0000","last_fullsized":"2024-03-24T11:32:41.816583+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:40.667849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:40.667849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:09.047430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":74,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.642774+0000","last_change":"2024-03-24T11:32:23.732096+0000","last_active":"2024-03-24T11:32:41.642774+0000","last_peered":"2024-03-24T11:32:41.642774+0000","last_clean":"2024-03-24T11:32:41.642774+0000","last_became_active":"2024-03-24T11:30:35.353571+0000","last_became_peered":"2024-03-24T11:30:35.353571+0000","last_unstale":"2024-03-24T11:32:41.642774+0000","last_undegraded":"2024-03-24T11:32:41.642774+0000","last_fullsized":"2024-03-24T11:32:41.642774+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:23.731967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.731967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:38.504397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":442,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.649104+0000","last_change":"2024-03-24T11:30:24.313618+0000","last_active":"2024-03-24T11:32:41.649104+0000","last_peered":"2024-03-24T11:32:41.649104+0000","last_clean":"2024-03-24T11:32:41.649104+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:32:41.649104+0000","last_undegraded":"2024-03-24T11:32:41.649104+0000","last_fullsized":"2024-03-24T11:32:41.649104+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:24.313152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.313152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"213'10","reported_seq":456,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643341+0000","last_change":"2024-03-24T11:29:30.032558+0000","last_active":"2024-03-24T11:32:41.643341+0000","last_peered":"2024-03-24T11:32:41.643341+0000","last_clean":"2024-03-24T11:32:41.643341+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:32:41.643341+0000","last_undegraded":"2024-03-24T11:32:41.643341+0000","last_fullsized":"2024-03-24T11:32:41.643341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:29:30.032476+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:29:30.032476+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":470,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.032010+0000","last_change":"2024-03-24T11:32:36.013375+0000","last_active":"2024-03-24T11:32:42.032010+0000","last_peered":"2024-03-24T11:32:42.032010+0000","last_clean":"2024-03-24T11:32:42.032010+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:32:42.032010+0000","last_undegraded":"2024-03-24T11:32:42.032010+0000","last_fullsized":"2024-03-24T11:32:42.032010+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.013287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.013287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:44.208803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":478,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.961485+0000","last_change":"2024-03-24T11:32:38.991091+0000","last_active":"2024-03-24T11:32:41.961485+0000","last_peered":"2024-03-24T11:32:41.961485+0000","last_clean":"2024-03-24T11:32:41.961485+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:32:41.961485+0000","last_undegraded":"2024-03-24T11:32:41.961485+0000","last_fullsized":"2024-03-24T11:32:41.961485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.991009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.991009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:05.887138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:43.213224+0000","last_change":"2024-03-24T11:32:43.213245+0000","last_active":"2024-03-24T11:32:43.213224+0000","last_peered":"2024-03-24T11:32:43.213224+0000","last_clean":"2024-03-24T11:32:43.213224+0000","last_became_active":"2024-03-24T11:30:35.353704+0000","last_became_peered":"2024-03-24T11:30:35.353704+0000","last_unstale":"2024-03-24T11:32:43.213224+0000","last_undegraded":"2024-03-24T11:32:43.213224+0000","last_fullsized":"2024-03-24T11:32:43.213224+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:43.213130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.213130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:06.338202+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"213'24","reported_seq":519,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.963185+0000","last_change":"2024-03-24T11:32:30.897616+0000","last_active":"2024-03-24T11:32:41.963185+0000","last_peered":"2024-03-24T11:32:41.963185+0000","last_clean":"2024-03-24T11:32:41.963185+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:32:41.963185+0000","last_undegraded":"2024-03-24T11:32:41.963185+0000","last_fullsized":"2024-03-24T11:32:41.963185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:30.897405+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.897405+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.068479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":454,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643260+0000","last_change":"2024-03-24T11:32:30.073961+0000","last_active":"2024-03-24T11:32:41.643260+0000","last_peered":"2024-03-24T11:32:41.643260+0000","last_clean":"2024-03-24T11:32:41.643260+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:32:41.643260+0000","last_undegraded":"2024-03-24T11:32:41.643260+0000","last_fullsized":"2024-03-24T11:32:41.643260+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:30.073873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.073873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:52.383142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.030766+0000","last_change":"2024-03-24T11:31:14.751432+0000","last_active":"2024-03-24T11:32:42.030766+0000","last_peered":"2024-03-24T11:32:42.030766+0000","last_clean":"2024-03-24T11:32:42.030766+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:32:42.030766+0000","last_undegraded":"2024-03-24T11:32:42.030766+0000","last_fullsized":"2024-03-24T11:32:42.030766+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.751309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.751309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:43.031712+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":466,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.770697+0000","last_change":"2024-03-24T11:31:41.853862+0000","last_active":"2024-03-24T11:32:41.770697+0000","last_peered":"2024-03-24T11:32:41.770697+0000","last_clean":"2024-03-24T11:32:41.770697+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:32:41.770697+0000","last_undegraded":"2024-03-24T11:32:41.770697+0000","last_fullsized":"2024-03-24T11:32:41.770697+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.853751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:04.343556+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"213'20","reported_seq":497,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031808+0000","last_change":"2024-03-24T11:32:25.904952+0000","last_active":"2024-03-24T11:32:42.031808+0000","last_peered":"2024-03-24T11:32:42.031808+0000","last_clean":"2024-03-24T11:32:42.031808+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:32:42.031808+0000","last_undegraded":"2024-03-24T11:32:42.031808+0000","last_fullsized":"2024-03-24T11:32:42.031808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:25.904851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.904851+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.755461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":479,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.764776+0000","last_change":"2024-03-24T11:31:37.672115+0000","last_active":"2024-03-24T11:32:41.764776+0000","last_peered":"2024-03-24T11:32:41.764776+0000","last_clean":"2024-03-24T11:32:41.764776+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:32:41.764776+0000","last_undegraded":"2024-03-24T11:32:41.764776+0000","last_fullsized":"2024-03-24T11:32:41.764776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:37.671989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.671989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.288597+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":660,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.963911+0000","last_change":"2024-03-24T11:32:41.963960+0000","last_active":"2024-03-24T11:32:41.963911+0000","last_peered":"2024-03-24T11:32:41.963911+0000","last_clean":"2024-03-24T11:32:41.963911+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:32:41.963911+0000","last_undegraded":"2024-03-24T11:32:41.963911+0000","last_fullsized":"2024-03-24T11:32:41.963911+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.963825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.606211+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":113,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815465+0000","last_change":"2024-03-24T11:30:35.355569+0000","last_active":"2024-03-24T11:32:41.815465+0000","last_peered":"2024-03-24T11:32:41.815465+0000","last_clean":"2024-03-24T11:32:41.815465+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:32:41.815465+0000","last_undegraded":"2024-03-24T11:32:41.815465+0000","last_fullsized":"2024-03-24T11:32:41.815465+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:30:22.784182+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:30:22.784182+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"213'18","reported_seq":474,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646688+0000","last_change":"2024-03-24T11:31:11.260142+0000","last_active":"2024-03-24T11:32:41.646688+0000","last_peered":"2024-03-24T11:32:41.646688+0000","last_clean":"2024-03-24T11:32:41.646688+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:32:41.646688+0000","last_undegraded":"2024-03-24T11:32:41.646688+0000","last_fullsized":"2024-03-24T11:32:41.646688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.259991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.259991+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":454,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643237+0000","last_change":"2024-03-24T11:30:11.883846+0000","last_active":"2024-03-24T11:32:41.643237+0000","last_peered":"2024-03-24T11:32:41.643237+0000","last_clean":"2024-03-24T11:32:41.643237+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:32:41.643237+0000","last_undegraded":"2024-03-24T11:32:41.643237+0000","last_fullsized":"2024-03-24T11:32:41.643237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:11.883599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.883599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.644361+0000","last_change":"2024-03-24T11:32:38.742586+0000","last_active":"2024-03-24T11:32:41.644361+0000","last_peered":"2024-03-24T11:32:41.644361+0000","last_clean":"2024-03-24T11:32:41.644361+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:32:41.644361+0000","last_undegraded":"2024-03-24T11:32:41.644361+0000","last_fullsized":"2024-03-24T11:32:41.644361+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.742478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.742478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.063731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.647378+0000","last_change":"2024-03-24T11:30:08.308986+0000","last_active":"2024-03-24T11:32:41.647378+0000","last_peered":"2024-03-24T11:32:41.647378+0000","last_clean":"2024-03-24T11:32:41.647378+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:32:41.647378+0000","last_undegraded":"2024-03-24T11:32:41.647378+0000","last_fullsized":"2024-03-24T11:32:41.647378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:08.308860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:08.308860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"213'27","reported_seq":517,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643926+0000","last_change":"2024-03-24T11:32:35.070335+0000","last_active":"2024-03-24T11:32:41.643926+0000","last_peered":"2024-03-24T11:32:41.643926+0000","last_clean":"2024-03-24T11:32:41.643926+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:32:41.643926+0000","last_undegraded":"2024-03-24T11:32:41.643926+0000","last_fullsized":"2024-03-24T11:32:41.643926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:32:35.070232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:35.070232+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.835251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031601+0000","last_change":"2024-03-24T11:31:10.852618+0000","last_active":"2024-03-24T11:32:42.031601+0000","last_peered":"2024-03-24T11:32:42.031601+0000","last_clean":"2024-03-24T11:32:42.031601+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:32:42.031601+0000","last_undegraded":"2024-03-24T11:32:42.031601+0000","last_fullsized":"2024-03-24T11:32:42.031601+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:10.852490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.852490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":470,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.808802+0000","last_change":"2024-03-24T11:32:42.808822+0000","last_active":"2024-03-24T11:32:42.808802+0000","last_peered":"2024-03-24T11:32:42.808802+0000","last_clean":"2024-03-24T11:32:42.808802+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:32:42.808802+0000","last_undegraded":"2024-03-24T11:32:42.808802+0000","last_fullsized":"2024-03-24T11:32:42.808802+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.808738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.808738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.742514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643952+0000","last_change":"2024-03-24T11:31:09.866310+0000","last_active":"2024-03-24T11:32:41.643952+0000","last_peered":"2024-03-24T11:32:41.643952+0000","last_clean":"2024-03-24T11:32:41.643952+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:32:41.643952+0000","last_undegraded":"2024-03-24T11:32:41.643952+0000","last_fullsized":"2024-03-24T11:32:41.643952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:09.866232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:31:09.866232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"213'7","reported_seq":471,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643134+0000","last_change":"2024-03-24T11:31:27.652238+0000","last_active":"2024-03-24T11:32:41.643134+0000","last_peered":"2024-03-24T11:32:41.643134+0000","last_clean":"2024-03-24T11:32:41.643134+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:32:41.643134+0000","last_undegraded":"2024-03-24T11:32:41.643134+0000","last_fullsized":"2024-03-24T11:32:41.643134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:31:27.652037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.652037+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815597+0000","last_change":"2024-03-24T11:30:14.299035+0000","last_active":"2024-03-24T11:32:41.815597+0000","last_peered":"2024-03-24T11:32:41.815597+0000","last_clean":"2024-03-24T11:32:41.815597+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:32:41.815597+0000","last_undegraded":"2024-03-24T11:32:41.815597+0000","last_fullsized":"2024-03-24T11:32:41.815597+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:14.298903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.298903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":74,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643376+0000","last_change":"2024-03-24T11:31:41.715026+0000","last_active":"2024-03-24T11:32:41.643376+0000","last_peered":"2024-03-24T11:32:41.643376+0000","last_clean":"2024-03-24T11:32:41.643376+0000","last_became_active":"2024-03-24T11:30:35.359239+0000","last_became_peered":"2024-03-24T11:30:35.359239+0000","last_unstale":"2024-03-24T11:32:41.643376+0000","last_undegraded":"2024-03-24T11:32:41.643376+0000","last_fullsized":"2024-03-24T11:32:41.643376+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:11.406185+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815663+0000","last_change":"2024-03-24T11:30:17.465431+0000","last_active":"2024-03-24T11:32:41.815663+0000","last_peered":"2024-03-24T11:32:41.815663+0000","last_clean":"2024-03-24T11:32:41.815663+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:32:41.815663+0000","last_undegraded":"2024-03-24T11:32:41.815663+0000","last_fullsized":"2024-03-24T11:32:41.815663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.465226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.465226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"213'15","reported_seq":475,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643194+0000","last_change":"2024-03-24T11:30:15.882584+0000","last_active":"2024-03-24T11:32:41.643194+0000","last_peered":"2024-03-24T11:32:41.643194+0000","last_clean":"2024-03-24T11:32:41.643194+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:32:41.643194+0000","last_undegraded":"2024-03-24T11:32:41.643194+0000","last_fullsized":"2024-03-24T11:32:41.643194+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:15.882446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:15.882446+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":470,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.816360+0000","last_change":"2024-03-24T11:31:41.687584+0000","last_active":"2024-03-24T11:32:41.816360+0000","last_peered":"2024-03-24T11:32:41.816360+0000","last_clean":"2024-03-24T11:32:41.816360+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:32:41.816360+0000","last_undegraded":"2024-03-24T11:32:41.816360+0000","last_fullsized":"2024-03-24T11:32:41.816360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.687474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.687474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:06.652087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.815357+0000","last_change":"2024-03-24T11:31:15.474018+0000","last_active":"2024-03-24T11:32:41.815357+0000","last_peered":"2024-03-24T11:32:41.815357+0000","last_clean":"2024-03-24T11:32:41.815357+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:32:41.815357+0000","last_undegraded":"2024-03-24T11:32:41.815357+0000","last_fullsized":"2024-03-24T11:32:41.815357+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.473909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.473909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031628+0000","last_change":"2024-03-24T11:31:18.759423+0000","last_active":"2024-03-24T11:32:42.031628+0000","last_peered":"2024-03-24T11:32:42.031628+0000","last_clean":"2024-03-24T11:32:42.031628+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:32:42.031628+0000","last_undegraded":"2024-03-24T11:32:42.031628+0000","last_fullsized":"2024-03-24T11:32:42.031628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:18.759165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.759165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"213'27","reported_seq":485,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.648790+0000","last_change":"2024-03-24T11:31:41.107651+0000","last_active":"2024-03-24T11:32:41.648790+0000","last_peered":"2024-03-24T11:32:41.648790+0000","last_clean":"2024-03-24T11:32:41.648790+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:32:41.648790+0000","last_undegraded":"2024-03-24T11:32:41.648790+0000","last_fullsized":"2024-03-24T11:32:41.648790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:31:41.107512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.107512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:44.858057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.646301+0000","last_change":"2024-03-24T11:30:23.137546+0000","last_active":"2024-03-24T11:32:41.646301+0000","last_peered":"2024-03-24T11:32:41.646301+0000","last_clean":"2024-03-24T11:32:41.646301+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:32:41.646301+0000","last_undegraded":"2024-03-24T11:32:41.646301+0000","last_fullsized":"2024-03-24T11:32:41.646301+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:23.137393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:23.137393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":454,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.642889+0000","last_change":"2024-03-24T11:32:24.692923+0000","last_active":"2024-03-24T11:32:41.642889+0000","last_peered":"2024-03-24T11:32:41.642889+0000","last_clean":"2024-03-24T11:32:41.642889+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:32:41.642889+0000","last_undegraded":"2024-03-24T11:32:41.642889+0000","last_fullsized":"2024-03-24T11:32:41.642889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.692844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.692844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.656723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.648937+0000","last_change":"2024-03-24T11:31:14.327893+0000","last_active":"2024-03-24T11:32:41.648937+0000","last_peered":"2024-03-24T11:32:41.648937+0000","last_clean":"2024-03-24T11:32:41.648937+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:32:41.648937+0000","last_undegraded":"2024-03-24T11:32:41.648937+0000","last_fullsized":"2024-03-24T11:32:41.648937+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.327777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.327777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"213'15","reported_seq":499,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.642823+0000","last_change":"2024-03-24T11:31:08.802687+0000","last_active":"2024-03-24T11:32:41.642823+0000","last_peered":"2024-03-24T11:32:41.642823+0000","last_clean":"2024-03-24T11:32:41.642823+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:32:41.642823+0000","last_undegraded":"2024-03-24T11:32:41.642823+0000","last_fullsized":"2024-03-24T11:32:41.642823+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:08.802538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.802538+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:42.031805+0000","last_change":"2024-03-24T11:32:35.031834+0000","last_active":"2024-03-24T11:32:42.031805+0000","last_peered":"2024-03-24T11:32:42.031805+0000","last_clean":"2024-03-24T11:32:42.031805+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:32:42.031805+0000","last_undegraded":"2024-03-24T11:32:42.031805+0000","last_fullsized":"2024-03-24T11:32:42.031805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:35.031735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:35.031735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":470,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.764704+0000","last_change":"2024-03-24T11:32:28.712422+0000","last_active":"2024-03-24T11:32:41.764704+0000","last_peered":"2024-03-24T11:32:41.764704+0000","last_clean":"2024-03-24T11:32:41.764704+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:32:41.764704+0000","last_undegraded":"2024-03-24T11:32:41.764704+0000","last_fullsized":"2024-03-24T11:32:41.764704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.712294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.712294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.891340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-24T11:32:41.643010+0000","last_change":"2024-03-24T11:31:07.885306+0000","last_active":"2024-03-24T11:32:41.643010+0000","last_peered":"2024-03-24T11:32:41.643010+0000","last_clean":"2024-03-24T11:32:41.643010+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:32:41.643010+0000","last_undegraded":"2024-03-24T11:32:41.643010+0000","last_fullsized":"2024-03-24T11:32:41.643010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:07.885177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:07.885177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":135,"num_read_kb":130,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070634,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491428,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253724,"statfs":{"total":95995035648,"available":95491813376,"internally_reserved":0,"allocated":745472,"data_stored":444015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":502462923},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364975,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":337688,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":336638,"kb_avail":93407464,"statfs":{"total":95995035648,"available":95649243136,"internally_reserved":0,"allocated":1060864,"data_stored":771916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":344717766},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515072,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93230080,"statfs":{"total":95995035648,"available":95467601920,"internally_reserved":0,"allocated":774144,"data_stored":445073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738574,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415404,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329748,"statfs":{"total":95995035648,"available":95569661952,"internally_reserved":0,"allocated":1081344,"data_stored":771450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411524,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334820,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410332,"statfs":{"total":95995035648,"available":95652179968,"internally_reserved":0,"allocated":1007616,"data_stored":706219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149885,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":479548,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93265604,"statfs":{"total":95995035648,"available":95503978496,"internally_reserved":0,"allocated":1363968,"data_stored":1034633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":489683399},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986361,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320688,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93424464,"statfs":{"total":95995035648,"available":95666651136,"internally_reserved":0,"allocated":757760,"data_stored":444443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659316,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491680,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253472,"statfs":{"total":95995035648,"available":95491555328,"internally_reserved":0,"allocated":1003520,"data_stored":706485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:32:50.155 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T11:32:50.155 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:32:50.155 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-24T11:32:50.155 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T11:32:50.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:50.223 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:50.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:50.348 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:50.450 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:50.473 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:50.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:50.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:50.635 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:50.678 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:50.691 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:32:50.691 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:32:50.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:50.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:50.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:50.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:50.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:50.977 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:50.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:51.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:51.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.103 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:51.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:51.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:51.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:51.322 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_3' created 2024-03-24T11:32:51.329 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:51.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:32:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.355 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:51.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:51.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:51.481 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:51.512 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:51 smithi179 ceph-mon[29608]: pgmap v585: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:32:51.512 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1742598983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T11:32:51.512 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:51 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T11:32:51.512 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:51 smithi179 ceph-mon[37907]: pgmap v585: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:32:51.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1742598983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T11:32:51.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:51 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T11:32:51.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:51.603 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:51.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:51 smithi181 ceph-mon[33998]: pgmap v585: 152 pgs: 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:32:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1742598983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T11:32:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:51 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T11:32:51.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:51.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:51.771 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:51.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:51.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:51.960 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:51.985 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:51.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:52.086 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:52.112 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:52.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:52.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:52.237 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:52.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:52.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:52.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:52.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T11:32:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[37907]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T11:32:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[29608]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:52.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:52 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:52.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:52.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:52.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:52.569 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:52 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T11:32:52.569 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:52 smithi181 ceph-mon[33998]: osdmap e267: 8 total, 8 up, 8 in 2024-03-24T11:32:52.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:52 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:32:52.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:52 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:32:52.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:52 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:32:52.570 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:52 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:32:52.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:52.613 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:52.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:52.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:52.740 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:52.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:52.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-24T11:32:52.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:52.893 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:52.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:52.995 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:53.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:53.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:53.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.146 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:53.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:53.247 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:53.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:53.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:53.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:53.424 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:32:53.424 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:32:53.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:53.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:53.589 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:53 smithi179 ceph-mon[29608]: pgmap v587: 168 pgs: 16 unknown, 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:32:53.589 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:53 smithi179 ceph-mon[29608]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T11:32:53.589 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:53 smithi179 ceph-mon[29608]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:32:53.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:53 smithi179 ceph-mon[37907]: pgmap v587: 168 pgs: 16 unknown, 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:32:53.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:53 smithi179 ceph-mon[37907]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T11:32:53.590 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:53 smithi179 ceph-mon[37907]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:32:53.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:53.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:53.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:53 smithi181 ceph-mon[33998]: pgmap v587: 168 pgs: 16 unknown, 152 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:32:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:53 smithi181 ceph-mon[33998]: osdmap e268: 8 total, 8 up, 8 in 2024-03-24T11:32:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:53 smithi181 ceph-mon[33998]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:32:53.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:53.772 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:53.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:53.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:53.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:53.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:54.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:54.062 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:32:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:32:54.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:32:54.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:54.148 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:54.250 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:54.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:54.375 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:54.376 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.401 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:54.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:54.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:54.525 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:54.593 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:54 smithi181 ceph-mon[33998]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T11:32:54.593 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:54.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:54.650 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:54 smithi179 ceph-mon[29608]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T11:32:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:54 smithi179 ceph-mon[37907]: osdmap e269: 8 total, 8 up, 8 in 2024-03-24T11:32:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:32:54.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.776 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:54.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:54.877 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:54.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:54.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:55.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:55.026 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:55.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:55.144 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:32:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:32:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:32:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.154 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:55.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:55.254 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:55.278 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:55.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:55.404 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:55.420 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:55 smithi179 ceph-mon[29608]: pgmap v590: 168 pgs: 2 unknown, 166 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:32:55.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:55 smithi179 ceph-mon[37907]: pgmap v590: 168 pgs: 2 unknown, 166 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:32:55.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:55.518 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:55 smithi181 ceph-mon[33998]: pgmap v590: 168 pgs: 2 unknown, 166 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:32:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.532 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:55.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:55.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:55.656 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "epoch": 269, 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "fsid": "6430538c-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "created": "2024-03-24T11:13:18.639983+0000", 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "modified": "2024-03-24T11:32:53.322561+0000", 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "last_up_change": "2024-03-24T11:18:40.079955+0000", 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "last_in_change": "2024-03-24T11:18:23.090476+0000", 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_num": 5801984, 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_set": [ 2024-03-24T11:32:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "noscrub", 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "pglog_hardlimit", 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "purged_snapdirs", 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "recovery_deletes", 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "sortbitwise" 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_version": 18, 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_snapshot": "", 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_max": 10, 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "max_osd": 8, 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "require_min_compat_client": "luminous", 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "min_compat_client": "luminous", 2024-03-24T11:32:55.675 INFO:teuthology.orchestra.run.smithi179.stdout: "require_osd_release": "squid", 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "allow_crimson": false, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "pools": [ 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 1, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": ".mgr", 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:16:40.018600+0000", 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:32:55.676 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 21, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 21, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 21, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 21, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 21, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:32:55.677 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "201", 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "199", 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:32:55.678 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:32:55.679 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "options": { 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_max": 32, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_min": 1 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.680 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "mgr": {} 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.6699999570846558, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.6699999570846558, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1.1399999856948853, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 1.8999999761581421, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 1.8999999761581421, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 0.875, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.681 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 2, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "datapool", 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:51.329095+0000", 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 8193, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:32:55.682 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 3, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 3, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 3, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 3, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 3, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.683 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "69", 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 2, 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 69, 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:32:55.684 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:32:55.685 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "rbd": {} 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:32:55.686 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 2.3299999237060547, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 2.3299999237060547, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 0.875, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 3, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": ".rgw.root", 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:52.018113+0000", 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:32:55.687 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:32:55.688 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "67", 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:32:55.689 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:32:55.690 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:32:55.691 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.75, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.75, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1.1399999856948853, 2024-03-24T11:32:55.692 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 2, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 2, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 0.875, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 4, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "default.rgw.log", 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:53.978924+0000", 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:32:55.693 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:32:55.694 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "69", 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:32:55.695 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:32:55.696 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:32:55.697 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.9700000286102295, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.9700000286102295, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1.1399999856948853, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 2.25, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 2.25, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 0.875, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-24T11:32:55.698 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 5, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "default.rgw.control", 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:55.925755+0000", 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:32:55.699 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:32:55.700 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "71", 2024-03-24T11:32:55.701 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:32:55.702 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:32:55.703 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:32:55.704 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.3200000524520874, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.3200000524520874, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1.1399999856948853, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 1.5, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 1.5, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 0.875, 2024-03-24T11:32:55.705 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:32:55.706 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-24T11:32:55.706 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.706 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 6, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "default.rgw.meta", 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:18:57.991833+0000", 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:32:55.707 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 32, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 32, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 32, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 32, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 32, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "73", 2024-03-24T11:32:55.708 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:32:55.709 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:32:55.710 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "options": { 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_bias": 4 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": { 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "rgw": {} 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:32:55.711 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 1.5399999618530273, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 1.5399999618530273, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1.1399999856948853, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 1.75, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 1.75, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 0.875, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "pool": 10, 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_name": "unique_pool_3", 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "create_time": "2024-03-24T11:32:51.091691+0000", 2024-03-24T11:32:55.712 INFO:teuthology.orchestra.run.smithi179.stdout: "flags": 1, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "flags_names": "hashpspool", 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "type": 1, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 3, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "min_size": 2, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_rule": 0, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_count": 0, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_target": 0, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "object_hash": 2, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_autoscale_mode": "off", 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num": 16, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num": 16, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_placement_num_target": 16, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_target": 16, 2024-03-24T11:32:55.713 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_num_pending": 16, 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "last_pg_merge_meta": { 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "source_pgid": "0.0", 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "ready_epoch": 0, 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_started": 0, 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "last_epoch_clean": 0, 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "source_version": "0'0", 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "target_version": "0'0" 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "last_change": "269", 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend": "0", 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T11:32:55.714 INFO:teuthology.orchestra.run.smithi179.stdout: "auid": 0, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_mode": "selfmanaged", 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_seq": 0, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "snap_epoch": 0, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "pool_snaps": [], 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps": "[]", 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_bytes": 0, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "quota_max_objects": 0, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "tiers": [], 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "tier_of": -1, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "read_tier": -1, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "write_tier": -1, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_mode": "none", 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_bytes": 0, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "target_max_objects": 0, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T11:32:55.715 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_flush_age": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "cache_min_evict_age": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profile": "", 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_params": { 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "none" 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_period": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_count": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "use_gmt_hitset": true, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "min_read_recency_for_promote": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "min_write_recency_for_promote": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "hit_set_search_last_n": 0, 2024-03-24T11:32:55.716 INFO:teuthology.orchestra.run.smithi179.stdout: "grade_table": [], 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "stripe_width": 0, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "expected_num_objects": 0, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "fast_read": false, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "options": {}, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "application_metadata": {}, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "read_balance": { 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "score_type": "Fair distribution", 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "score_acting": 2.630000114440918, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "score_stable": 2.630000114440918, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "optimal_score": 1.1399999856948853, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_acting": 3, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "raw_score_stable": 3, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity_weighted": 0.875, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity": 1, 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-24T11:32:55.717 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "osds": [ 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 0, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "14f300f0-7498-4ead-a87e-d9f38bdff21b", 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 8, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 267, 2024-03-24T11:32:55.718 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6802", 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6803", 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.719 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6804", 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6805", 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.720 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6808", 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6809", 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.721 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6806", 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6807", 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 1725503377 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6803/1725503377", 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6805/1725503377", 2024-03-24T11:32:55.722 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6809/1725503377", 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6807/1725503377", 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 1, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667", 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.723 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 13, 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 267, 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6810", 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.724 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6811", 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6812", 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6813", 2024-03-24T11:32:55.725 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6816", 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6817", 2024-03-24T11:32:55.726 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6814", 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6815", 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2564847275 2024-03-24T11:32:55.727 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.728 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.728 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.728 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6811/2564847275", 2024-03-24T11:32:55.728 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6813/2564847275", 2024-03-24T11:32:55.728 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6817/2564847275", 2024-03-24T11:32:55.728 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6815/2564847275", 2024-03-24T11:32:55.729 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.729 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.729 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.729 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.729 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.729 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 2, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "863b92b8-c68c-41a3-aed4-fd79e96d34dc", 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 18, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 267, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.730 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6818", 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6819", 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.731 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6820", 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6821", 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.732 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6824", 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6825", 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6822", 2024-03-24T11:32:55.733 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6823", 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3297713842 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6819/3297713842", 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6821/3297713842", 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6825/3297713842", 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6823/3297713842", 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.734 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 3, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "c3d6b564-195d-4771-9aae-93a9e2eaba52", 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 26, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 237, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.735 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6826", 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6827", 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.736 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6828", 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.737 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6829", 2024-03-24T11:32:55.738 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.738 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.738 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.738 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.738 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.738 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.738 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6832", 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6833", 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.739 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6830", 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:6831", 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 944142471 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.179:6827/944142471", 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.179:6829/944142471", 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.179:6833/944142471", 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.179:6831/944142471", 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.740 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 4, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "91bf5303-3a94-407d-aef1-e43bf0e16176", 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 31, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 267, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.741 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6800", 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6801", 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.742 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6802", 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6803", 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.743 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6806", 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6807", 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.744 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6804", 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6805", 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 3231660415 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6801/3231660415", 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6803/3231660415", 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6807/3231660415", 2024-03-24T11:32:55.745 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6805/3231660415", 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 5, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "cba3f489-e8be-46f7-a281-e69a0ad08646", 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.746 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 41, 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 267, 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6808", 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6809", 2024-03-24T11:32:55.747 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6810", 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6811", 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.748 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6814", 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6815", 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.749 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6812", 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6813", 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 775575782 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.750 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6809/775575782", 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6811/775575782", 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6815/775575782", 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6813/775575782", 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 6, 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "44a2c3a0-b704-4c10-a257-c54de47ee793", 2024-03-24T11:32:55.751 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 1, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 50, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 267, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.752 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6816", 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6817", 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6818", 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.753 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6819", 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6822", 2024-03-24T11:32:55.754 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6823", 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6820", 2024-03-24T11:32:55.755 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6821", 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 2357509374 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6817/2357509374", 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6819/2357509374", 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6823/2357509374", 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6821/2357509374", 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.756 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 7, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "uuid": "df60c4b4-d2f2-4e1f-8e1f-3169c403de00", 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "up": 1, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "in": 1, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "weight": 1, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_affinity": 0, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_begin": 0, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "last_clean_end": 0, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "up_from": 59, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "up_thru": 211, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "down_at": 0, 2024-03-24T11:32:55.757 INFO:teuthology.orchestra.run.smithi179.stdout: "lost_at": 0, 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addrs": { 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6824", 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6825", 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.758 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addrs": { 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6826", 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6827", 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.759 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addrs": { 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6830", 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6831", 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.760 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addrs": { 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "addrvec": [ 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v2", 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6828", 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.181:6829", 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 732039117 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.761 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "public_addr": "172.21.15.181:6825/732039117", 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "cluster_addr": "172.21.15.181:6827/732039117", 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_back_addr": "172.21.15.181:6831/732039117", 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "heartbeat_front_addr": "172.21.15.181:6829/732039117", 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "state": [ 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "exists", 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "up" 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_xinfo": [ 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.762 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 0, 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:32:10.600588+0000", 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 1, 2024-03-24T11:32:55.763 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:32:04.936491+0000", 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 2, 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.764 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:32:16.457081+0000", 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 3, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:32:04.356041+0000", 2024-03-24T11:32:55.765 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 4, 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:32:09.867948+0000", 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.766 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 5, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:31:46.771642+0000", 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 6, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.767 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:32:23.260425+0000", 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "osd": 7, 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "down_stamp": "0.000000", 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_probability": 0, 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "laggy_interval": 0, 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "features": 4540701547738038271, 2024-03-24T11:32:55.768 INFO:teuthology.orchestra.run.smithi179.stdout: "old_weight": 0, 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "last_purged_snaps_scrub": "2024-03-24T11:32:12.495983+0000", 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "dead_epoch": 0 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_upmap": [], 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_upmap_items": [ 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "pgid": "3.2", 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "mappings": [ 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "from": 6, 2024-03-24T11:32:55.769 INFO:teuthology.orchestra.run.smithi179.stdout: "to": 7 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: "pgid": "4.7", 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: "mappings": [ 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: "from": 1, 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: "to": 2 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.770 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "pgid": "4.d", 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "mappings": [ 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "from": 1, 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "to": 5 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "pgid": "4.18", 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "mappings": [ 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "from": 1, 2024-03-24T11:32:55.771 INFO:teuthology.orchestra.run.smithi179.stdout: "to": 2 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_upmap_primaries": [], 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "pg_temp": [], 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "primary_temp": [], 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "blocklist": { 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/1519226585": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6800/2865046672": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2472348284": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2047439415": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6801/3169437": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:32:55.772 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6800/3169437": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6801/243537818": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/3185072343": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/3344305045": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2748363950": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6800/243537818": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2066930919": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/2108875546": "2024-03-25T11:14:17.688945+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:6801/2865046672": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/4087802319": "2024-03-25T11:19:31.961858+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/883221702": "2024-03-25T11:13:51.373673+0000", 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "172.21.15.179:0/4255117010": "2024-03-25T11:19:31.961858+0000" 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "range_blocklist": {}, 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "erasure_code_profiles": { 2024-03-24T11:32:55.773 INFO:teuthology.orchestra.run.smithi179.stdout: "default": { 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "crush-failure-domain": "osd", 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "k": "2", 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "m": "1", 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "plugin": "jerasure", 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "technique": "reed_sol_van" 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "removed_snaps_queue": [], 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "new_removed_snaps": [], 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "new_purged_snaps": [], 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "crush_node_flags": {}, 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "device_class_flags": {}, 2024-03-24T11:32:55.774 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_mode": { 2024-03-24T11:32:55.775 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_mode_enabled": false, 2024-03-24T11:32:55.775 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_bucket_count": 0, 2024-03-24T11:32:55.775 INFO:teuthology.orchestra.run.smithi179.stdout: "degraded_stretch_mode": 0, 2024-03-24T11:32:55.775 INFO:teuthology.orchestra.run.smithi179.stdout: "recovering_stretch_mode": 0, 2024-03-24T11:32:55.775 INFO:teuthology.orchestra.run.smithi179.stdout: "stretch_mode_bucket": 0 2024-03-24T11:32:55.775 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:32:55.775 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:32:55.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:55.799 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:55.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:55.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:55.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:56.025 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:56.048 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:56.105 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:32:56.105 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:32:56.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.177 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:56.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:56.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:56.301 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:56.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:56 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2848899036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:32:56.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:56.414 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:32:56.414 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:32:56.420 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:32:56.426 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:56.440 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:32:56.459 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:32:56.479 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:32:56.498 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:32:56.517 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:32:56.518 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:32:56.518 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:32:56.518 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:32:56.519 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:32:56.519 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:32:56.519 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:32:56.519 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:32:56.519 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:32:56.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:56.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:56.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:56.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:56 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2848899036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:32:56.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:56 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2848899036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T11:32:56.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T11:32:56.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:56.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:56.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:56.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:56.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:56.986 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:57.030 INFO:tasks.thrashosds.thrasher:j is {'epoch': 269, 'fsid': '6430538c-e9cf-11ee-95cd-87774f69a715', 'created': '2024-03-24T11:13:18.639983+0000', 'modified': '2024-03-24T11:32:53.322561+0000', 'last_up_change': '2024-03-24T11:18:40.079955+0000', 'last_in_change': '2024-03-24T11:18:23.090476+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:16:40.018600+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '199', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6699999570846558, 'score_stable': 1.6699999570846558, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T11:18:51.329095+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T11:18:52.018113+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T11:18:53.978924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T11:18:55.925755+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T11:18:57.991833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-24T11:32:51.091691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '269', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '14f300f0-7498-4ead-a87e-d9f38bdff21b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6802', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6803', 'nonce': 1725503377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6804', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6805', 'nonce': 1725503377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6808', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6809', 'nonce': 1725503377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6806', 'nonce': 1725503377}, {'type': 'v1', 'addr': '172.21.15.179:6807', 'nonce': 1725503377}]}, 'public_addr': '172.21.15.179:6803/1725503377', 'cluster_addr': '172.21.15.179:6805/1725503377', 'heartbeat_back_addr': '172.21.15.179:6809/1725503377', 'heartbeat_front_addr': '172.21.15.179:6807/1725503377', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6810', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6811', 'nonce': 2564847275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6812', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6813', 'nonce': 2564847275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6816', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6817', 'nonce': 2564847275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6814', 'nonce': 2564847275}, {'type': 'v1', 'addr': '172.21.15.179:6815', 'nonce': 2564847275}]}, 'public_addr': '172.21.15.179:6811/2564847275', 'cluster_addr': '172.21.15.179:6813/2564847275', 'heartbeat_back_addr': '172.21.15.179:6817/2564847275', 'heartbeat_front_addr': '172.21.15.179:6815/2564847275', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '863b92b8-c68c-41a3-aed4-fd79e96d34dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6818', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6819', 'nonce': 3297713842}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6820', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6821', 'nonce': 3297713842}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6824', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6825', 'nonce': 3297713842}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6822', 'nonce': 3297713842}, {'type': 'v1', 'addr': '172.21.15.179:6823', 'nonce': 3297713842}]}, 'public_addr': '172.21.15.179:6819/3297713842', 'cluster_addr': '172.21.15.179:6821/3297713842', 'heartbeat_back_addr': '172.21.15.179:6825/3297713842', 'heartbeat_front_addr': '172.21.15.179:6823/3297713842', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c3d6b564-195d-4771-9aae-93a9e2eaba52', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 237, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6826', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6827', 'nonce': 944142471}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6828', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6829', 'nonce': 944142471}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6832', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6833', 'nonce': 944142471}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6830', 'nonce': 944142471}, {'type': 'v1', 'addr': '172.21.15.179:6831', 'nonce': 944142471}]}, 'public_addr': '172.21.15.179:6827/944142471', 'cluster_addr': '172.21.15.179:6829/944142471', 'heartbeat_back_addr': '172.21.15.179:6833/944142471', 'heartbeat_front_addr': '172.21.15.179:6831/944142471', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91bf5303-3a94-407d-aef1-e43bf0e16176', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6801', 'nonce': 3231660415}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6803', 'nonce': 3231660415}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6807', 'nonce': 3231660415}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 3231660415}, {'type': 'v1', 'addr': '172.21.15.181:6805', 'nonce': 3231660415}]}, 'public_addr': '172.21.15.181:6801/3231660415', 'cluster_addr': '172.21.15.181:6803/3231660415', 'heartbeat_back_addr': '172.21.15.181:6807/3231660415', 'heartbeat_front_addr': '172.21.15.181:6805/3231660415', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cba3f489-e8be-46f7-a281-e69a0ad08646', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6809', 'nonce': 775575782}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6811', 'nonce': 775575782}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6815', 'nonce': 775575782}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 775575782}, {'type': 'v1', 'addr': '172.21.15.181:6813', 'nonce': 775575782}]}, 'public_addr': '172.21.15.181:6809/775575782', 'cluster_addr': '172.21.15.181:6811/775575782', 'heartbeat_back_addr': '172.21.15.181:6815/775575782', 'heartbeat_front_addr': '172.21.15.181:6813/775575782', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '44a2c3a0-b704-4c10-a257-c54de47ee793', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6816', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6817', 'nonce': 2357509374}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6818', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6819', 'nonce': 2357509374}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6822', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6823', 'nonce': 2357509374}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6820', 'nonce': 2357509374}, {'type': 'v1', 'addr': '172.21.15.181:6821', 'nonce': 2357509374}]}, 'public_addr': '172.21.15.181:6817/2357509374', 'cluster_addr': '172.21.15.181:6819/2357509374', 'heartbeat_back_addr': '172.21.15.181:6823/2357509374', 'heartbeat_front_addr': '172.21.15.181:6821/2357509374', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'df60c4b4-d2f2-4e1f-8e1f-3169c403de00', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 211, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6824', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6825', 'nonce': 732039117}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6826', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6827', 'nonce': 732039117}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6830', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6831', 'nonce': 732039117}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6828', 'nonce': 732039117}, {'type': 'v1', 'addr': '172.21.15.181:6829', 'nonce': 732039117}]}, 'public_addr': '172.21.15.181:6825/732039117', 'cluster_addr': '172.21.15.181:6827/732039117', 'heartbeat_back_addr': '172.21.15.181:6831/732039117', 'heartbeat_front_addr': '172.21.15.181:6829/732039117', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:32:10.600588+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:32:04.936491+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:32:16.457081+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:32:04.356041+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:32:09.867948+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:31:46.771642+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:32:23.260425+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T11:32:12.495983+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.2', 'mappings': [{'from': 6, 'to': 7}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.179:0/1519226585': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:6800/2865046672': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/2472348284': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/2047439415': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:6801/3169437': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:6800/3169437': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:6801/243537818': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:0/3185072343': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/3344305045': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:0/2748363950': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:6800/243537818': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:0/2066930919': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:0/2108875546': '2024-03-25T11:14:17.688945+0000', '172.21.15.179:6801/2865046672': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/4087802319': '2024-03-25T11:19:31.961858+0000', '172.21.15.179:0/883221702': '2024-03-25T11:13:51.373673+0000', '172.21.15.179:0/4255117010': '2024-03-25T11:19:31.961858+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T11:32:57.031 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:32:57.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:57.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:57.164 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:57.189 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:57.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:57.315 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:57.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:57 smithi181 ceph-mon[33998]: pgmap v591: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:32:57.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4208904133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:57.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:57.440 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:57.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:57.565 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:57.630 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:57 smithi179 ceph-mon[29608]: pgmap v591: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:32:57.630 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4208904133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:57.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:57 smithi179 ceph-mon[37907]: pgmap v591: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:32:57.630 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4208904133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:32:57.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:57.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:57.769 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:57.788 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:32:57.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:57.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:57.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:32:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:57.948 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:57.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:32:58.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:58.073 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:58.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:58.174 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:58.198 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:58.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:58.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:58.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:58.323 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:58.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:58.352 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:32:58.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4032851652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:58.398 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:58 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:58.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:58.449 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:58.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:58.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:58.565 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4032851652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:58.566 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:58 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:58.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4032851652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:58.566 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:58 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:32:58.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:58.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:58.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:58.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:32:58.702 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:58.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:32:58.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:58.812 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:32:58.812 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:32:58.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:32:58.812 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:154485 7.7 7:e9fcc1e6:::benchmark_data_smithi179_78267_object35291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:23:15.106344+0000", 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 583.70458109499998, 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10214155499999999, 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 154485 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106344+0000", 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106344+0000", 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106345+0000", 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.31e-06 2024-03-24T11:32:58.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106350+0000", 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7500000000000003e-06 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106360+0000", 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9990000000000003e-06 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106367+0000", 2024-03-24T11:32:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5590000000000002e-06 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106423+0000", 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6175999999999999e-05 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106560+0000", 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013683799999999999 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.106697+0000", 2024-03-24T11:32:58.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013766000000000001 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.109419+0000", 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027217090000000001 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.167559+0000", 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.058139569000000002 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.208332+0000", 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.040773729000000002 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.208440+0000", 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107908 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:23:15.208486+0000", 2024-03-24T11:32:58.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5348e-05 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:216695 7.7 7:ebf3756e:::benchmark_data_smithi179_78267_object97501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:18.997488+0000", 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.81343695999999, 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.137933637, 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.819 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 216695 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997488+0000", 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997488+0000", 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997489+0000", 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7000000000000004e-07 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997493+0000", 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5700000000000001e-06 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997504+0000", 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1017e-05 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997511+0000", 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.939e-06 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997562+0000", 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0949000000000003e-05 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997693+0000", 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000131851 2024-03-24T11:32:58.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:18.997824+0000", 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000130576 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.000524+0000", 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027003880000000002 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.006166+0000", 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056413540000000003 2024-03-24T11:32:58.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.135350+0000", 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.129183984 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.135388+0000", 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7880999999999999e-05 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.135422+0000", 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4357999999999999e-05 2024-03-24T11:32:58.824 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:216704 7.7 7:ebbea3bd:::benchmark_data_smithi179_78267_object97510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:19.007159+0000", 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.80376580300003, 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.145395316, 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 216704 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.825 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007159+0000", 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007159+0000", 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007160+0000", 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8800000000000002e-07 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007164+0000", 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.331e-06 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007171+0000", 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5429999999999996e-06 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007176+0000", 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2109999999999997e-06 2024-03-24T11:32:58.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007225+0000", 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9054999999999997e-05 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007366+0000", 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014058899999999999 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,3", 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.007851+0000", 2024-03-24T11:32:58.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048472499999999998 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.011045+0000", 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031943459999999998 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.135461+0000", 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.124415807 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.152481+0000", 2024-03-24T11:32:58.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017019995 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.152521+0000", 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0107000000000001e-05 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.152555+0000", 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3819000000000003e-05 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24698.0:216712 7.d 7:b23f2af7:::benchmark_data_smithi179_78267_object97518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:24:19.013242+0000", 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 519.79768377100004, 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.126256232, 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24698", 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/2501259558", 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 216712 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013242+0000", 2024-03-24T11:32:58.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013242+0000", 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013242+0000", 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3699999999999999e-07 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013246+0000", 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8380000000000004e-06 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013257+0000", 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0533000000000001e-05 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013264+0000", 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6760000000000007e-06 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013318+0000", 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3164000000000001e-05 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013393+0000", 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5809000000000006e-05 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.013459+0000", 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5726000000000001e-05 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.015982+0000", 2024-03-24T11:32:58.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025225040000000001 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.016995+0000", 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001013321 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.139403+0000", 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.12240838 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.139455+0000", 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1748e-05 2024-03-24T11:32:58.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:24:19.139498+0000", 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2595999999999997e-05 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:164407 8.9 8:9a903af1:::benchmark_data_smithi179_105658_object36365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.311417+0000", 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 343.49950811999997, 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10670746, 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.837 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164407 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311417+0000", 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311417+0000", 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311418+0000", 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.035e-06 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311422+0000", 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0929999999999996e-06 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311435+0000", 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.273e-05 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311442+0000", 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0620000000000003e-06 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311524+0000", 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1469000000000002e-05 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311627+0000", 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010395000000000001 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.311736+0000", 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108515 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.314700+0000", 2024-03-24T11:32:58.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029641670000000002 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.316852+0000", 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021516059999999999 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.418041+0000", 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101189717 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.418088+0000", 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6071e-05 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.418125+0000", 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7045000000000002e-05 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:165003 8.9 8:95c9f60b:::benchmark_data_smithi179_105658_object36961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:15.933374+0000", 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 342.87755096400002, 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.117111048, 2024-03-24T11:32:58.843 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165003 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933374+0000", 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933374+0000", 2024-03-24T11:32:58.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933376+0000", 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8759999999999999e-06 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933382+0000", 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4330000000000003e-06 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933399+0000", 2024-03-24T11:32:58.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7127999999999999e-05 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933409+0000", 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9080000000000007e-06 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933479+0000", 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0722999999999994e-05 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933627+0000", 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014713500000000001 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.933760+0000", 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013344499999999999 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.938968+0000", 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0052082200000000004 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:15.939273+0000", 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030475199999999999 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.050383+0000", 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111110232 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.050438+0000", 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5108999999999997e-05 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.050485+0000", 2024-03-24T11:32:58.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7086999999999998e-05 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:165181 8.9 8:936bf970:::benchmark_data_smithi179_105658_object37139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:16.168490+0000", 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 342.64243496400002, 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.117235991, 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.849 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165181 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.168490+0000", 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.168490+0000", 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.168491+0000", 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8899999999999999e-07 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.168495+0000", 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6239999999999999e-06 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.168504+0000", 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0829999999999993e-06 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.168511+0000", 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1099999999999997e-06 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.172374+0000", 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003863165 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.172476+0000", 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000101888 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.172580+0000", 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103965 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.175551+0000", 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029713169999999998 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.177814+0000", 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022625689999999999 2024-03-24T11:32:58.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.285644+0000", 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107829936 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.285676+0000", 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2230000000000001e-05 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.285726+0000", 2024-03-24T11:32:58.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0414999999999998e-05 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:165201 8.9 8:9acb0d18:::benchmark_data_smithi179_105658_object37159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:16.185881+0000", 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 342.62504447499998, 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107907739, 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:58.855 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165201 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.185881+0000", 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.185881+0000", 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.185881+0000", 2024-03-24T11:32:58.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1699999999999998e-07 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.185884+0000", 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9790000000000002e-06 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.185892+0000", 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9740000000000002e-06 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.185899+0000", 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4980000000000003e-06 2024-03-24T11:32:58.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.190825+0000", 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049265410000000004 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.190900+0000", 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4552e-05 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.190969+0000", 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9047000000000005e-05 2024-03-24T11:32:58.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.192708+0000", 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017387150000000001 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.192887+0000", 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017960200000000001 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.293656+0000", 2024-03-24T11:32:58.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100769157 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.293715+0000", 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8881000000000001e-05 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.293789+0000", 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3176000000000005e-05 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:165767 8.9 8:97d61d6f:::benchmark_data_smithi179_105658_object37725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:16.797248+0000", 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 342.01367707000003, 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101121905, 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165767 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.797248+0000", 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.797248+0000", 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.797250+0000", 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.787e-06 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.797257+0000", 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3989999999999998e-06 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.797268+0000", 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0665e-05 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.797276+0000", 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4560000000000003e-06 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.798170+0000", 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089455999999999997 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.798283+0000", 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112521 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.798408+0000", 2024-03-24T11:32:58.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012499700000000001 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.803753+0000", 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053457840000000001 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.810046+0000", 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062921940000000001 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.898268+0000", 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088222421999999995 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.898325+0000", 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6604999999999998e-05 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.898370+0000", 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5515e-05 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.866 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:165780 8.9 8:944c4fd7:::benchmark_data_smithi179_105658_object37738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:27:16.805470+0000", 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 342.005455047, 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107445473, 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165780 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805470+0000", 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805470+0000", 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805472+0000", 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2729999999999999e-06 2024-03-24T11:32:58.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805476+0000", 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6330000000000004e-06 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805491+0000", 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.499e-05 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805499+0000", 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6550000000000004e-06 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805548+0000", 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9054000000000002e-05 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805657+0000", 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109592 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.805773+0000", 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011566900000000001 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.809424+0000", 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036511640000000001 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.810081+0000", 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000656282 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.912767+0000", 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10268632 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.912872+0000", 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104657 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:27:16.912916+0000", 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4184000000000003e-05 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.24959.0:226251 8.9 8:92b537b6:::benchmark_data_smithi179_105658_object98209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:28:08.874559+0000", 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 289.93636617800001, 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11431843799999999, 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:32:58.873 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.24959", 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 226251 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874559+0000", 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874559+0000", 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874560+0000", 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2359999999999999e-06 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874565+0000", 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0130000000000001e-06 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874578+0000", 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2186e-05 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874585+0000", 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.204e-06 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874631+0000", 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6173e-05 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874727+0000", 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6403999999999994e-05 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:32:58.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.874803+0000", 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5675999999999998e-05 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.877700+0000", 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028970129999999999 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.877750+0000", 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0291999999999999e-05 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.988806+0000", 2024-03-24T11:32:58.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111056134 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.988841+0000", 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4597000000000001e-05 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:28:08.988878+0000", 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6510000000000001e-05 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.878 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:209274 9.0 e248/211 9:0251295c:::benchmark_data_smithi179_130919_object75687:head v 248'13090, mlcod=248'13088)", 2024-03-24T11:32:58.879 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:37.129330+0000", 2024-03-24T11:32:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:58.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:58.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.681595341000005, 2024-03-24T11:32:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010944341, 2024-03-24T11:32:58.880 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129330+0000", 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129330+0000", 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129331+0000", 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1999999999999998e-07 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129337+0000", 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3090000000000001e-06 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129348+0000", 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0720000000000001e-05 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129355+0000", 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2239999999999998e-06 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129413+0000", 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7883000000000001e-05 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.129432+0000", 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9179000000000001e-05 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.140166+0000", 2024-03-24T11:32:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010733827 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:37.140274+0000", 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108379 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:213753 9.b e248/211 9:dadd811f:::benchmark_data_smithi179_130919_object80166:head v 248'13361, mlcod=248'13359)", 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:40.389403+0000", 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 78.421522173, 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012950691, 2024-03-24T11:32:58.884 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389403+0000", 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389403+0000", 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389404+0000", 2024-03-24T11:32:58.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0999999999999998e-07 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389409+0000", 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6579999999999999e-06 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389421+0000", 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1705000000000001e-05 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389428+0000", 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3980000000000004e-06 2024-03-24T11:32:58.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389627+0000", 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000199755 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.389683+0000", 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6047000000000001e-05 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.402251+0000", 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012567337 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.402354+0000", 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103081 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:213756 9.b e248/211 9:df823278:::benchmark_data_smithi179_130919_object80169:head v 248'13362, mlcod=248'13359)", 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:40.392832+0000", 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 78.418093529000004, 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011379604999999999, 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.888 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.392832+0000", 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.392832+0000", 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.392832+0000", 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8700000000000005e-07 2024-03-24T11:32:58.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.392838+0000", 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9719999999999997e-06 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.392849+0000", 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0626e-05 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.392856+0000", 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6730000000000003e-06 2024-03-24T11:32:58.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.397020+0000", 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041638300000000003 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.397067+0000", 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7484000000000001e-05 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.404164+0000", 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0070968890000000003 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:40.404211+0000", 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7444000000000002e-05 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:215705 9.b e249/211 9:d48132c2:::benchmark_data_smithi179_130919_object82118:head v 249'13500, mlcod=249'13496)", 2024-03-24T11:32:58.892 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:42.084545+0000", 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 76.726380543000005, 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010702579, 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084545+0000", 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084545+0000", 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084545+0000", 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5700000000000002e-07 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084551+0000", 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.003e-06 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084568+0000", 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6359999999999999e-05 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084575+0000", 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2049999999999996e-06 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084662+0000", 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7224000000000004e-05 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.084685+0000", 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2374e-05 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.095193+0000", 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010507864 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.095247+0000", 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4792e-05 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:249174 9.b e256/211 9:d04d6671:::benchmark_data_smithi179_130919_object115587:head v 256'15563, mlcod=256'15561)", 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:32:10.245179+0000", 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.565746068000003, 2024-03-24T11:32:58.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010603598000000001, 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245179+0000", 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245179+0000", 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245180+0000", 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.114e-06 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245189+0000", 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0259999999999996e-06 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245210+0000", 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0693e-05 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245219+0000", 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2159999999999995e-06 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245428+0000", 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000208347 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.245479+0000", 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1808999999999998e-05 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.255734+0000", 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010254898 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:10.255783+0000", 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8495000000000002e-05 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:251996 9.b e258/211 9:da8c742c:::benchmark_data_smithi179_130919_object118409:head v 258'15737, mlcod=258'15734)", 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:32:12.390042+0000", 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.420883214, 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011132634000000001, 2024-03-24T11:32:58.900 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390042+0000", 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390042+0000", 2024-03-24T11:32:58.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390043+0000", 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2500000000000004e-07 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390049+0000", 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0109999999999997e-06 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390064+0000", 2024-03-24T11:32:58.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4895999999999999e-05 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390071+0000", 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5170000000000002e-06 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390156+0000", 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4932000000000006e-05 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.390176+0000", 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9962999999999998e-05 2024-03-24T11:32:58.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.401031+0000", 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010855201 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:12.401175+0000", 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000143289 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.904 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:254058 9.b e259/211 9:d2722bd3:::benchmark_data_smithi179_130919_object120471:head v 259'15870, mlcod=259'15867)", 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:32:13.981413+0000", 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.829512573000002, 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010231707, 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981413+0000", 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981413+0000", 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981414+0000", 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7999999999999993e-07 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981420+0000", 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6449999999999999e-06 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981440+0000", 2024-03-24T11:32:58.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.003e-05 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981448+0000", 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2130000000000002e-06 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981523+0000", 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5699999999999997e-05 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.981543+0000", 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0227000000000001e-05 2024-03-24T11:32:58.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.991559+0000", 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010015822000000001 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:13.991644+0000", 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.509e-05 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:261006 9.b e261/211 9:d3880426:::benchmark_data_smithi179_130919_object127419:head v 261'16300, mlcod=261'16295)", 2024-03-24T11:32:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:32:19.287852+0000", 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.523073081, 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011890866999999999, 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287852+0000", 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287852+0000", 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287853+0000", 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.68e-07 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287858+0000", 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0799999999999996e-06 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287868+0000", 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9680000000000001e-06 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287874+0000", 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5409999999999997e-06 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287949+0000", 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4771999999999995e-05 2024-03-24T11:32:58.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.287979+0000", 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9646e-05 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.299693+0000", 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.01171433 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:32:19.299743+0000", 2024-03-24T11:32:58.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9861999999999998e-05 2024-03-24T11:32:58.913 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.913 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.913 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.913 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:32:58.913 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:32:58.913 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:32:58.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.005 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:59.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:59.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:32:59.129 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:32:59.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:32:59.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:32:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:32:59.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:59.380 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:59 smithi181 ceph-mon[33998]: pgmap v592: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-24T11:32:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:59 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:59 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:59.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:32:59 smithi181 ceph-mon[33998]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T11:32:59.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:32:59.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:32:59.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:32:59.632 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:32:59.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T11:32:59.695 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[29608]: pgmap v592: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-24T11:32:59.695 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:59.695 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:59.695 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[29608]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T11:32:59.695 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[37907]: pgmap v592: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-24T11:32:59.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:32:59.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:32:59.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:32:59 smithi179 ceph-mon[37907]: osdmap e270: 8 total, 8 up, 8 in 2024-03-24T11:32:59.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:32:59.758 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:32:59.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:32:59.887 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:32:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:32:59.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:32:59.919 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:32:59.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.015 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:00.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:00.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:00.141 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:00.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:00.267 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:00.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:33:00.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:00.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:00.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:00.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:00.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:00.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:00.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:00.769 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:00.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:00.894 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:00.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:00.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:01.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:01.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:01.149 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:01.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:01.275 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:01.354 INFO:teuthology.orchestra.run.smithi179.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-24T11:33:01.377 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:01.402 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:01.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:01.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:01.614 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:01 smithi181 ceph-mon[33998]: pgmap v594: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:33:01.614 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1182918476' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:33:01.631 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:01.655 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:01.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:01 smithi179 ceph-mon[29608]: pgmap v594: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:33:01.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1182918476' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:33:01.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:01 smithi179 ceph-mon[37907]: pgmap v594: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:33:01.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1182918476' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:33:01.662 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:33:01.663 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:33:01.663 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-03-24T11:33:01.663 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:33:01.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:01.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:01.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:01.907 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:01.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:01.984 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:33:01.988 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:33:02.008 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.032 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:02.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:02.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:02.157 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:02.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:02.282 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:02.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T11:33:02.384 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:02.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:02 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1182918476' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:33:02.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:02 smithi181 ceph-mon[33998]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T11:33:02.408 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:02.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:02.533 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:02.625 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:02 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1182918476' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:33:02.625 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:02 smithi179 ceph-mon[29608]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T11:33:02.625 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:02 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1182918476' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:33:02.625 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:02 smithi179 ceph-mon[37907]: osdmap e271: 8 total, 8 up, 8 in 2024-03-24T11:33:02.632 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:33:02.634 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:02.658 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:02.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:02.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:02.886 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-03-24T11:33:01.348031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1300,"num_read_kb":1052,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3512,"ondisk_log_size":3512,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3365904,"kb_used_data":8020,"kb_used_omap":12,"kb_used_meta":3357747,"kb_avail":746595312,"statfs":{"total":767960285184,"available":764513599488,"internally_reserved":0,"allocated":8212480,"data_stored":5603508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3438333489},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.020788"},"pg_stats":[{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007048+0000","last_change":"2024-03-24T11:32:52.328191+0000","last_active":"2024-03-24T11:32:59.007048+0000","last_peered":"2024-03-24T11:32:59.007048+0000","last_clean":"2024-03-24T11:32:59.007048+0000","last_became_active":"2024-03-24T11:32:52.327879+0000","last_became_peered":"2024-03-24T11:32:52.327879+0000","last_unstale":"2024-03-24T11:32:59.007048+0000","last_undegraded":"2024-03-24T11:32:59.007048+0000","last_fullsized":"2024-03-24T11:32:59.007048+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.314171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"213'20","reported_seq":497,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329408+0000","last_change":"2024-03-24T11:32:36.799202+0000","last_active":"2024-03-24T11:32:53.329408+0000","last_peered":"2024-03-24T11:32:53.329408+0000","last_clean":"2024-03-24T11:32:53.329408+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:32:53.329408+0000","last_undegraded":"2024-03-24T11:32:53.329408+0000","last_fullsized":"2024-03-24T11:32:53.329408+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:36.799084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.799084+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:06.179015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":457,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346041+0000","last_change":"2024-03-24T11:29:27.048143+0000","last_active":"2024-03-24T11:32:58.346041+0000","last_peered":"2024-03-24T11:32:58.346041+0000","last_clean":"2024-03-24T11:32:58.346041+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:32:58.346041+0000","last_undegraded":"2024-03-24T11:32:58.346041+0000","last_fullsized":"2024-03-24T11:32:58.346041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:27.048023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:29:27.048023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":666,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347567+0000","last_change":"2024-03-24T11:32:37.091705+0000","last_active":"2024-03-24T11:32:58.347567+0000","last_peered":"2024-03-24T11:32:58.347567+0000","last_clean":"2024-03-24T11:32:58.347567+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:32:58.347567+0000","last_undegraded":"2024-03-24T11:32:58.347567+0000","last_fullsized":"2024-03-24T11:32:58.347567+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:37.091630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.091630+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.785943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":459,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329782+0000","last_change":"2024-03-24T11:31:15.842164+0000","last_active":"2024-03-24T11:32:53.329782+0000","last_peered":"2024-03-24T11:32:53.329782+0000","last_clean":"2024-03-24T11:32:53.329782+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:32:53.329782+0000","last_undegraded":"2024-03-24T11:32:53.329782+0000","last_fullsized":"2024-03-24T11:32:53.329782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.842054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.842054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":519,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345602+0000","last_change":"2024-03-24T11:31:10.289542+0000","last_active":"2024-03-24T11:32:58.345602+0000","last_peered":"2024-03-24T11:32:58.345602+0000","last_clean":"2024-03-24T11:32:58.345602+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:32:58.345602+0000","last_undegraded":"2024-03-24T11:32:58.345602+0000","last_fullsized":"2024-03-24T11:32:58.345602+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:10.289309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.289309+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.348250+0000","last_change":"2024-03-24T11:32:52.671692+0000","last_active":"2024-03-24T11:32:58.348250+0000","last_peered":"2024-03-24T11:32:58.348250+0000","last_clean":"2024-03-24T11:32:58.348250+0000","last_became_active":"2024-03-24T11:32:52.671360+0000","last_became_peered":"2024-03-24T11:32:52.671360+0000","last_unstale":"2024-03-24T11:32:58.348250+0000","last_undegraded":"2024-03-24T11:32:58.348250+0000","last_fullsized":"2024-03-24T11:32:58.348250+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:18.513514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"213'39","reported_seq":556,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329948+0000","last_change":"2024-03-24T11:32:42.683845+0000","last_active":"2024-03-24T11:32:53.329948+0000","last_peered":"2024-03-24T11:32:53.329948+0000","last_clean":"2024-03-24T11:32:53.329948+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:32:53.329948+0000","last_undegraded":"2024-03-24T11:32:53.329948+0000","last_fullsized":"2024-03-24T11:32:53.329948+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:32:42.683738+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.683738+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.142199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":502,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940221+0000","last_change":"2024-03-24T11:32:23.906538+0000","last_active":"2024-03-24T11:32:53.940221+0000","last_peered":"2024-03-24T11:32:53.940221+0000","last_clean":"2024-03-24T11:32:53.940221+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:32:53.940221+0000","last_undegraded":"2024-03-24T11:32:53.940221+0000","last_fullsized":"2024-03-24T11:32:53.940221+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:32:23.906455+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.906455+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.906085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":481,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.008349+0000","last_change":"2024-03-24T11:32:25.619271+0000","last_active":"2024-03-24T11:32:59.008349+0000","last_peered":"2024-03-24T11:32:59.008349+0000","last_clean":"2024-03-24T11:32:59.008349+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:32:59.008349+0000","last_undegraded":"2024-03-24T11:32:59.008349+0000","last_fullsized":"2024-03-24T11:32:59.008349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:25.619184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.619184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.019241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":477,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347307+0000","last_change":"2024-03-24T11:31:35.151699+0000","last_active":"2024-03-24T11:32:58.347307+0000","last_peered":"2024-03-24T11:32:58.347307+0000","last_clean":"2024-03-24T11:32:58.347307+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:32:58.347307+0000","last_undegraded":"2024-03-24T11:32:58.347307+0000","last_fullsized":"2024-03-24T11:32:58.347307+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:35.151566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:35.151566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":615,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464662+0000","last_change":"2024-03-24T11:31:31.621753+0000","last_active":"2024-03-24T11:32:58.464662+0000","last_peered":"2024-03-24T11:32:58.464662+0000","last_clean":"2024-03-24T11:32:58.464662+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:32:58.464662+0000","last_undegraded":"2024-03-24T11:32:58.464662+0000","last_fullsized":"2024-03-24T11:32:58.464662+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.621646+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.866195+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940361+0000","last_change":"2024-03-24T11:32:52.324098+0000","last_active":"2024-03-24T11:32:53.940361+0000","last_peered":"2024-03-24T11:32:53.940361+0000","last_clean":"2024-03-24T11:32:53.940361+0000","last_became_active":"2024-03-24T11:32:52.323807+0000","last_became_peered":"2024-03-24T11:32:52.323807+0000","last_unstale":"2024-03-24T11:32:53.940361+0000","last_undegraded":"2024-03-24T11:32:53.940361+0000","last_fullsized":"2024-03-24T11:32:53.940361+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:57.895418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"213'19","reported_seq":506,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329664+0000","last_change":"2024-03-24T11:32:37.784109+0000","last_active":"2024-03-24T11:32:53.329664+0000","last_peered":"2024-03-24T11:32:53.329664+0000","last_clean":"2024-03-24T11:32:53.329664+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:32:53.329664+0000","last_undegraded":"2024-03-24T11:32:53.329664+0000","last_fullsized":"2024-03-24T11:32:53.329664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:37.784011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.784011+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:39.413955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":471,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328757+0000","last_change":"2024-03-24T11:31:29.730215+0000","last_active":"2024-03-24T11:32:53.328757+0000","last_peered":"2024-03-24T11:32:53.328757+0000","last_clean":"2024-03-24T11:32:53.328757+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:32:53.328757+0000","last_undegraded":"2024-03-24T11:32:53.328757+0000","last_fullsized":"2024-03-24T11:32:53.328757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.730078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.730078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:54.570128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":465,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465550+0000","last_change":"2024-03-24T11:30:10.252564+0000","last_active":"2024-03-24T11:32:58.465550+0000","last_peered":"2024-03-24T11:32:58.465550+0000","last_clean":"2024-03-24T11:32:58.465550+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:32:58.465550+0000","last_undegraded":"2024-03-24T11:32:58.465550+0000","last_fullsized":"2024-03-24T11:32:58.465550+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:10.252339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:30:10.252339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":445,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346195+0000","last_change":"2024-03-24T11:29:39.057894+0000","last_active":"2024-03-24T11:32:58.346195+0000","last_peered":"2024-03-24T11:32:58.346195+0000","last_clean":"2024-03-24T11:32:58.346195+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:32:58.346195+0000","last_undegraded":"2024-03-24T11:32:58.346195+0000","last_fullsized":"2024-03-24T11:32:58.346195+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:39.057644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:39.057644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":634,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007615+0000","last_change":"2024-03-24T11:32:24.651691+0000","last_active":"2024-03-24T11:32:59.007615+0000","last_peered":"2024-03-24T11:32:59.007615+0000","last_clean":"2024-03-24T11:32:59.007615+0000","last_became_active":"2024-03-24T11:28:43.190009+0000","last_became_peered":"2024-03-24T11:28:43.190009+0000","last_unstale":"2024-03-24T11:32:59.007615+0000","last_undegraded":"2024-03-24T11:32:59.007615+0000","last_fullsized":"2024-03-24T11:32:59.007615+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:24.651587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.651587+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.416321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328591+0000","last_change":"2024-03-24T11:32:52.671340+0000","last_active":"2024-03-24T11:32:53.328591+0000","last_peered":"2024-03-24T11:32:53.328591+0000","last_clean":"2024-03-24T11:32:53.328591+0000","last_became_active":"2024-03-24T11:32:52.671000+0000","last_became_peered":"2024-03-24T11:32:52.671000+0000","last_unstale":"2024-03-24T11:32:53.328591+0000","last_undegraded":"2024-03-24T11:32:53.328591+0000","last_fullsized":"2024-03-24T11:32:53.328591+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:14.443024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"213'34","reported_seq":535,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.008507+0000","last_change":"2024-03-24T11:32:30.766638+0000","last_active":"2024-03-24T11:32:59.008507+0000","last_peered":"2024-03-24T11:32:59.008507+0000","last_clean":"2024-03-24T11:32:59.008507+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:32:59.008507+0000","last_undegraded":"2024-03-24T11:32:59.008507+0000","last_fullsized":"2024-03-24T11:32:59.008507+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:32:30.766549+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.766549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.554399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":465,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345551+0000","last_change":"2024-03-24T11:31:22.167292+0000","last_active":"2024-03-24T11:32:58.345551+0000","last_peered":"2024-03-24T11:32:58.345551+0000","last_clean":"2024-03-24T11:32:58.345551+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:32:58.345551+0000","last_undegraded":"2024-03-24T11:32:58.345551+0000","last_fullsized":"2024-03-24T11:32:58.345551+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.167171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.167171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":640,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329780+0000","last_change":"2024-03-24T11:32:18.783487+0000","last_active":"2024-03-24T11:32:53.329780+0000","last_peered":"2024-03-24T11:32:53.329780+0000","last_clean":"2024-03-24T11:32:53.329780+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:32:53.329780+0000","last_undegraded":"2024-03-24T11:32:53.329780+0000","last_fullsized":"2024-03-24T11:32:53.329780+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:18.783339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.783339+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:29.190654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":461,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465161+0000","last_change":"2024-03-24T11:31:28.690070+0000","last_active":"2024-03-24T11:32:58.465161+0000","last_peered":"2024-03-24T11:32:58.465161+0000","last_clean":"2024-03-24T11:32:58.465161+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:32:58.465161+0000","last_undegraded":"2024-03-24T11:32:58.465161+0000","last_fullsized":"2024-03-24T11:32:58.465161+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.689946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.689946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614958+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":606,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464705+0000","last_change":"2024-03-24T11:30:11.263503+0000","last_active":"2024-03-24T11:32:58.464705+0000","last_peered":"2024-03-24T11:32:58.464705+0000","last_clean":"2024-03-24T11:32:58.464705+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:32:58.464705+0000","last_undegraded":"2024-03-24T11:32:58.464705+0000","last_fullsized":"2024-03-24T11:32:58.464705+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:30:11.263251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.263251+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329473+0000","last_change":"2024-03-24T11:32:52.325471+0000","last_active":"2024-03-24T11:32:53.329473+0000","last_peered":"2024-03-24T11:32:53.329473+0000","last_clean":"2024-03-24T11:32:53.329473+0000","last_became_active":"2024-03-24T11:32:52.325161+0000","last_became_peered":"2024-03-24T11:32:52.325161+0000","last_unstale":"2024-03-24T11:32:53.329473+0000","last_undegraded":"2024-03-24T11:32:53.329473+0000","last_fullsized":"2024-03-24T11:32:53.329473+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:09.514137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":471,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328737+0000","last_change":"2024-03-24T11:32:39.715836+0000","last_active":"2024-03-24T11:32:53.328737+0000","last_peered":"2024-03-24T11:32:53.328737+0000","last_clean":"2024-03-24T11:32:53.328737+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:32:53.328737+0000","last_undegraded":"2024-03-24T11:32:53.328737+0000","last_fullsized":"2024-03-24T11:32:53.328737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.715759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.083140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"213'30","reported_seq":504,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345653+0000","last_change":"2024-03-24T11:31:31.099444+0000","last_active":"2024-03-24T11:32:58.345653+0000","last_peered":"2024-03-24T11:32:58.345653+0000","last_clean":"2024-03-24T11:32:58.345653+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:32:58.345653+0000","last_undegraded":"2024-03-24T11:32:58.345653+0000","last_fullsized":"2024-03-24T11:32:58.345653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:31:31.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.099209+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":457,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347580+0000","last_change":"2024-03-24T11:29:28.019634+0000","last_active":"2024-03-24T11:32:58.347580+0000","last_peered":"2024-03-24T11:32:58.347580+0000","last_clean":"2024-03-24T11:32:58.347580+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:32:58.347580+0000","last_undegraded":"2024-03-24T11:32:58.347580+0000","last_fullsized":"2024-03-24T11:32:58.347580+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:28.018667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:29:28.018667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":451,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328910+0000","last_change":"2024-03-24T11:30:17.941383+0000","last_active":"2024-03-24T11:32:53.328910+0000","last_peered":"2024-03-24T11:32:53.328910+0000","last_clean":"2024-03-24T11:32:53.328910+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:32:53.328910+0000","last_undegraded":"2024-03-24T11:32:53.328910+0000","last_fullsized":"2024-03-24T11:32:53.328910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.941259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":618,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329559+0000","last_change":"2024-03-24T11:31:25.692917+0000","last_active":"2024-03-24T11:32:53.329559+0000","last_peered":"2024-03-24T11:32:53.329559+0000","last_clean":"2024-03-24T11:32:53.329559+0000","last_became_active":"2024-03-24T11:28:38.173086+0000","last_became_peered":"2024-03-24T11:28:38.173086+0000","last_unstale":"2024-03-24T11:32:53.329559+0000","last_undegraded":"2024-03-24T11:32:53.329559+0000","last_fullsized":"2024-03-24T11:32:53.329559+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.692790+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:28:38.972499+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.692790+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941197+0000","last_change":"2024-03-24T11:32:52.673849+0000","last_active":"2024-03-24T11:32:53.941197+0000","last_peered":"2024-03-24T11:32:53.941197+0000","last_clean":"2024-03-24T11:32:53.941197+0000","last_became_active":"2024-03-24T11:32:52.673505+0000","last_became_peered":"2024-03-24T11:32:52.673505+0000","last_unstale":"2024-03-24T11:32:53.941197+0000","last_undegraded":"2024-03-24T11:32:53.941197+0000","last_fullsized":"2024-03-24T11:32:53.941197+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:00.873085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":627,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345677+0000","last_change":"2024-03-24T11:31:08.290147+0000","last_active":"2024-03-24T11:32:58.345677+0000","last_peered":"2024-03-24T11:32:58.345677+0000","last_clean":"2024-03-24T11:32:58.345677+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:32:58.345677+0000","last_undegraded":"2024-03-24T11:32:58.345677+0000","last_fullsized":"2024-03-24T11:32:58.345677+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:08.289960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.289960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"213'28","reported_seq":509,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328846+0000","last_change":"2024-03-24T11:32:34.830458+0000","last_active":"2024-03-24T11:32:53.328846+0000","last_peered":"2024-03-24T11:32:53.328846+0000","last_clean":"2024-03-24T11:32:53.328846+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:32:53.328846+0000","last_undegraded":"2024-03-24T11:32:53.328846+0000","last_fullsized":"2024-03-24T11:32:53.328846+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:32:34.830322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.830322+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.916433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":482,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007780+0000","last_change":"2024-03-24T11:32:40.035107+0000","last_active":"2024-03-24T11:32:59.007780+0000","last_peered":"2024-03-24T11:32:59.007780+0000","last_clean":"2024-03-24T11:32:59.007780+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:32:59.007780+0000","last_undegraded":"2024-03-24T11:32:59.007780+0000","last_fullsized":"2024-03-24T11:32:59.007780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:32:40.035022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.035022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:47.238292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":91,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940914+0000","last_change":"2024-03-24T11:32:33.957761+0000","last_active":"2024-03-24T11:32:53.940914+0000","last_peered":"2024-03-24T11:32:53.940914+0000","last_clean":"2024-03-24T11:32:53.940914+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:32:53.940914+0000","last_undegraded":"2024-03-24T11:32:53.940914+0000","last_fullsized":"2024-03-24T11:32:53.940914+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:33.957632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.957632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.301310+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":612,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346093+0000","last_change":"2024-03-24T11:31:18.144752+0000","last_active":"2024-03-24T11:32:58.346093+0000","last_peered":"2024-03-24T11:32:58.346093+0000","last_clean":"2024-03-24T11:32:58.346093+0000","last_became_active":"2024-03-24T11:28:38.174888+0000","last_became_peered":"2024-03-24T11:28:38.174888+0000","last_unstale":"2024-03-24T11:32:58.346093+0000","last_undegraded":"2024-03-24T11:32:58.346093+0000","last_fullsized":"2024-03-24T11:32:58.346093+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.144322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.144322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329298+0000","last_change":"2024-03-24T11:32:52.670936+0000","last_active":"2024-03-24T11:32:53.329298+0000","last_peered":"2024-03-24T11:32:53.329298+0000","last_clean":"2024-03-24T11:32:53.329298+0000","last_became_active":"2024-03-24T11:32:52.670591+0000","last_became_peered":"2024-03-24T11:32:52.670591+0000","last_unstale":"2024-03-24T11:32:53.329298+0000","last_undegraded":"2024-03-24T11:32:53.329298+0000","last_fullsized":"2024-03-24T11:32:53.329298+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:05.733726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":652,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007924+0000","last_change":"2024-03-24T11:32:42.033121+0000","last_active":"2024-03-24T11:32:59.007924+0000","last_peered":"2024-03-24T11:32:59.007924+0000","last_clean":"2024-03-24T11:32:59.007924+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:32:59.007924+0000","last_undegraded":"2024-03-24T11:32:59.007924+0000","last_fullsized":"2024-03-24T11:32:59.007924+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.033019+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.550557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"213'20","reported_seq":515,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007311+0000","last_change":"2024-03-24T11:32:21.588381+0000","last_active":"2024-03-24T11:32:59.007311+0000","last_peered":"2024-03-24T11:32:59.007311+0000","last_clean":"2024-03-24T11:32:59.007311+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:32:59.007311+0000","last_undegraded":"2024-03-24T11:32:59.007311+0000","last_fullsized":"2024-03-24T11:32:59.007311+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:21.588239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.588239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.355301+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":484,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465397+0000","last_change":"2024-03-24T11:31:06.261980+0000","last_active":"2024-03-24T11:32:58.465397+0000","last_peered":"2024-03-24T11:32:58.465397+0000","last_clean":"2024-03-24T11:32:58.465397+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:32:58.465397+0000","last_undegraded":"2024-03-24T11:32:58.465397+0000","last_fullsized":"2024-03-24T11:32:58.465397+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:06.261883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.261883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614817+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":462,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006567+0000","last_change":"2024-03-24T11:32:22.994060+0000","last_active":"2024-03-24T11:32:59.006567+0000","last_peered":"2024-03-24T11:32:59.006567+0000","last_clean":"2024-03-24T11:32:59.006567+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:32:59.006567+0000","last_undegraded":"2024-03-24T11:32:59.006567+0000","last_fullsized":"2024-03-24T11:32:59.006567+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:32:22.993851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:22.993851+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.104574+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":631,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941403+0000","last_change":"2024-03-24T11:31:42.062272+0000","last_active":"2024-03-24T11:32:53.941403+0000","last_peered":"2024-03-24T11:32:53.941403+0000","last_clean":"2024-03-24T11:32:53.941403+0000","last_became_active":"2024-03-24T11:28:38.178260+0000","last_became_peered":"2024-03-24T11:28:38.178260+0000","last_unstale":"2024-03-24T11:32:53.941403+0000","last_undegraded":"2024-03-24T11:32:53.941403+0000","last_fullsized":"2024-03-24T11:32:53.941403+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.062151+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:10.775672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940901+0000","last_change":"2024-03-24T11:32:52.327929+0000","last_active":"2024-03-24T11:32:53.940901+0000","last_peered":"2024-03-24T11:32:53.940901+0000","last_clean":"2024-03-24T11:32:53.940901+0000","last_became_active":"2024-03-24T11:32:52.327652+0000","last_became_peered":"2024-03-24T11:32:52.327652+0000","last_unstale":"2024-03-24T11:32:53.940901+0000","last_undegraded":"2024-03-24T11:32:53.940901+0000","last_fullsized":"2024-03-24T11:32:53.940901+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:12.978546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":487,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940706+0000","last_change":"2024-03-24T11:31:33.075345+0000","last_active":"2024-03-24T11:32:53.940706+0000","last_peered":"2024-03-24T11:32:53.940706+0000","last_clean":"2024-03-24T11:32:53.940706+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:32:53.940706+0000","last_undegraded":"2024-03-24T11:32:53.940706+0000","last_fullsized":"2024-03-24T11:32:53.940706+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:33.075228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.075228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:56.584947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"213'29","reported_seq":520,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007745+0000","last_change":"2024-03-24T11:32:26.907881+0000","last_active":"2024-03-24T11:32:59.007745+0000","last_peered":"2024-03-24T11:32:59.007745+0000","last_clean":"2024-03-24T11:32:59.007745+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:32:59.007745+0000","last_undegraded":"2024-03-24T11:32:59.007745+0000","last_fullsized":"2024-03-24T11:32:59.007745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:32:26.907769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.907769+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:32.644339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":489,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.008205+0000","last_change":"2024-03-24T11:31:25.294292+0000","last_active":"2024-03-24T11:32:59.008205+0000","last_peered":"2024-03-24T11:32:59.008205+0000","last_clean":"2024-03-24T11:32:59.008205+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:32:59.008205+0000","last_undegraded":"2024-03-24T11:32:59.008205+0000","last_fullsized":"2024-03-24T11:32:59.008205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:25.294054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.294054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.987800+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":477,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006733+0000","last_change":"2024-03-24T11:32:45.851666+0000","last_active":"2024-03-24T11:32:59.006733+0000","last_peered":"2024-03-24T11:32:59.006733+0000","last_clean":"2024-03-24T11:32:59.006733+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:32:59.006733+0000","last_undegraded":"2024-03-24T11:32:59.006733+0000","last_fullsized":"2024-03-24T11:32:59.006733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.851566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.851566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:02.916651+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":631,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941029+0000","last_change":"2024-03-24T11:31:18.894161+0000","last_active":"2024-03-24T11:32:53.941029+0000","last_peered":"2024-03-24T11:32:53.941029+0000","last_clean":"2024-03-24T11:32:53.941029+0000","last_became_active":"2024-03-24T11:28:38.174742+0000","last_became_peered":"2024-03-24T11:28:38.174742+0000","last_unstale":"2024-03-24T11:32:53.941029+0000","last_undegraded":"2024-03-24T11:32:53.941029+0000","last_fullsized":"2024-03-24T11:32:53.941029+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.894050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.894050+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.939573+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006807+0000","last_change":"2024-03-24T11:32:52.671042+0000","last_active":"2024-03-24T11:32:59.006807+0000","last_peered":"2024-03-24T11:32:59.006807+0000","last_clean":"2024-03-24T11:32:59.006807+0000","last_became_active":"2024-03-24T11:32:52.670737+0000","last_became_peered":"2024-03-24T11:32:52.670737+0000","last_unstale":"2024-03-24T11:32:59.006807+0000","last_undegraded":"2024-03-24T11:32:59.006807+0000","last_fullsized":"2024-03-24T11:32:59.006807+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:03.192188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":465,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465563+0000","last_change":"2024-03-24T11:30:27.391730+0000","last_active":"2024-03-24T11:32:58.465563+0000","last_peered":"2024-03-24T11:32:58.465563+0000","last_clean":"2024-03-24T11:32:58.465563+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:32:58.465563+0000","last_undegraded":"2024-03-24T11:32:58.465563+0000","last_fullsized":"2024-03-24T11:32:58.465563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:30:27.391607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614765+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"213'24","reported_seq":505,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007742+0000","last_change":"2024-03-24T11:32:21.972798+0000","last_active":"2024-03-24T11:32:59.007742+0000","last_peered":"2024-03-24T11:32:59.007742+0000","last_clean":"2024-03-24T11:32:59.007742+0000","last_became_active":"2024-03-24T11:18:55.681266+0000","last_became_peered":"2024-03-24T11:18:55.681266+0000","last_unstale":"2024-03-24T11:32:59.007742+0000","last_undegraded":"2024-03-24T11:32:59.007742+0000","last_fullsized":"2024-03-24T11:32:59.007742+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:21.972644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.972644+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.486043+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":280,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006700+0000","last_change":"2024-03-24T11:32:39.822026+0000","last_active":"2024-03-24T11:32:59.006700+0000","last_peered":"2024-03-24T11:32:59.006700+0000","last_clean":"2024-03-24T11:32:59.006700+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:32:59.006700+0000","last_undegraded":"2024-03-24T11:32:59.006700+0000","last_fullsized":"2024-03-24T11:32:59.006700+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:32:39.821933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.821933+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.501549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":145,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":476,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007123+0000","last_change":"2024-03-24T11:31:21.629091+0000","last_active":"2024-03-24T11:32:59.007123+0000","last_peered":"2024-03-24T11:32:59.007123+0000","last_clean":"2024-03-24T11:32:59.007123+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:32:59.007123+0000","last_undegraded":"2024-03-24T11:32:59.007123+0000","last_fullsized":"2024-03-24T11:32:59.007123+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:21.628978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:21.628978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070430+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":477,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006955+0000","last_change":"2024-03-24T11:31:42.720752+0000","last_active":"2024-03-24T11:32:59.006955+0000","last_peered":"2024-03-24T11:32:59.006955+0000","last_clean":"2024-03-24T11:32:59.006955+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:32:59.006955+0000","last_undegraded":"2024-03-24T11:32:59.006955+0000","last_fullsized":"2024-03-24T11:32:59.006955+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:42.720504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.720504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:00.102566+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":628,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007727+0000","last_change":"2024-03-24T11:32:26.629982+0000","last_active":"2024-03-24T11:32:59.007727+0000","last_peered":"2024-03-24T11:32:59.007727+0000","last_clean":"2024-03-24T11:32:59.007727+0000","last_became_active":"2024-03-24T11:28:38.173843+0000","last_became_peered":"2024-03-24T11:28:38.173843+0000","last_unstale":"2024-03-24T11:32:59.007727+0000","last_undegraded":"2024-03-24T11:32:59.007727+0000","last_fullsized":"2024-03-24T11:32:59.007727+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:26.629894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.629894+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.767066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940736+0000","last_change":"2024-03-24T11:32:52.328051+0000","last_active":"2024-03-24T11:32:53.940736+0000","last_peered":"2024-03-24T11:32:53.940736+0000","last_clean":"2024-03-24T11:32:53.940736+0000","last_became_active":"2024-03-24T11:32:52.327768+0000","last_became_peered":"2024-03-24T11:32:52.327768+0000","last_unstale":"2024-03-24T11:32:53.940736+0000","last_undegraded":"2024-03-24T11:32:53.940736+0000","last_fullsized":"2024-03-24T11:32:53.940736+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:08.212172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":93,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.348542+0000","last_change":"2024-03-24T11:32:41.207872+0000","last_active":"2024-03-24T11:32:58.348542+0000","last_peered":"2024-03-24T11:32:58.348542+0000","last_clean":"2024-03-24T11:32:58.348542+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:32:58.348542+0000","last_undegraded":"2024-03-24T11:32:58.348542+0000","last_fullsized":"2024-03-24T11:32:58.348542+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:41.207792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.207792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.669676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"213'36","reported_seq":708,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941032+0000","last_change":"2024-03-24T11:32:25.853007+0000","last_active":"2024-03-24T11:32:53.941032+0000","last_peered":"2024-03-24T11:32:53.941032+0000","last_clean":"2024-03-24T11:32:53.941032+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:32:53.941032+0000","last_undegraded":"2024-03-24T11:32:53.941032+0000","last_fullsized":"2024-03-24T11:32:53.941032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'36","last_scrub_stamp":"2024-03-24T11:32:25.852913+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.852913+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.114595+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":471,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345414+0000","last_change":"2024-03-24T11:32:28.089047+0000","last_active":"2024-03-24T11:32:58.345414+0000","last_peered":"2024-03-24T11:32:58.345414+0000","last_clean":"2024-03-24T11:32:58.345414+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:32:58.345414+0000","last_undegraded":"2024-03-24T11:32:58.345414+0000","last_fullsized":"2024-03-24T11:32:58.345414+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.088969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.088969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.266746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":381,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006646+0000","last_change":"2024-03-24T11:31:07.885631+0000","last_active":"2024-03-24T11:32:59.006646+0000","last_peered":"2024-03-24T11:32:59.006646+0000","last_clean":"2024-03-24T11:32:59.006646+0000","last_became_active":"2024-03-24T11:30:35.355009+0000","last_became_peered":"2024-03-24T11:30:35.355009+0000","last_unstale":"2024-03-24T11:32:59.006646+0000","last_undegraded":"2024-03-24T11:32:59.006646+0000","last_fullsized":"2024-03-24T11:32:59.006646+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:07.885508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:07.885508+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070412+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":477,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465379+0000","last_change":"2024-03-24T11:31:26.676287+0000","last_active":"2024-03-24T11:32:58.465379+0000","last_peered":"2024-03-24T11:32:58.465379+0000","last_clean":"2024-03-24T11:32:58.465379+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:32:58.465379+0000","last_undegraded":"2024-03-24T11:32:58.465379+0000","last_fullsized":"2024-03-24T11:32:58.465379+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:26.676209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:31:26.676209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614833+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464607+0000","last_change":"2024-03-24T11:32:52.671396+0000","last_active":"2024-03-24T11:32:58.464607+0000","last_peered":"2024-03-24T11:32:58.464607+0000","last_clean":"2024-03-24T11:32:58.464607+0000","last_became_active":"2024-03-24T11:32:52.671052+0000","last_became_peered":"2024-03-24T11:32:52.671052+0000","last_unstale":"2024-03-24T11:32:58.464607+0000","last_undegraded":"2024-03-24T11:32:58.464607+0000","last_fullsized":"2024-03-24T11:32:58.464607+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.002020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":481,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.348461+0000","last_change":"2024-03-24T11:31:31.994930+0000","last_active":"2024-03-24T11:32:58.348461+0000","last_peered":"2024-03-24T11:32:58.348461+0000","last_clean":"2024-03-24T11:32:58.348461+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:32:58.348461+0000","last_undegraded":"2024-03-24T11:32:58.348461+0000","last_fullsized":"2024-03-24T11:32:58.348461+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:31.994604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.994604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"213'17","reported_seq":493,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347089+0000","last_change":"2024-03-24T11:32:38.137085+0000","last_active":"2024-03-24T11:32:58.347089+0000","last_peered":"2024-03-24T11:32:58.347089+0000","last_clean":"2024-03-24T11:32:58.347089+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:32:58.347089+0000","last_undegraded":"2024-03-24T11:32:58.347089+0000","last_fullsized":"2024-03-24T11:32:58.347089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'17","last_scrub_stamp":"2024-03-24T11:32:38.136966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.136966+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.504065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":498,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346993+0000","last_change":"2024-03-24T11:32:18.855130+0000","last_active":"2024-03-24T11:32:58.346993+0000","last_peered":"2024-03-24T11:32:58.346993+0000","last_clean":"2024-03-24T11:32:58.346993+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:32:58.346993+0000","last_undegraded":"2024-03-24T11:32:58.346993+0000","last_fullsized":"2024-03-24T11:32:58.346993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:18.854979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.854979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:20.771742+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":490,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006864+0000","last_change":"2024-03-24T11:31:29.443192+0000","last_active":"2024-03-24T11:32:59.006864+0000","last_peered":"2024-03-24T11:32:59.006864+0000","last_clean":"2024-03-24T11:32:59.006864+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:32:59.006864+0000","last_undegraded":"2024-03-24T11:32:59.006864+0000","last_fullsized":"2024-03-24T11:32:59.006864+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.443013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.443013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":91,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941090+0000","last_change":"2024-03-24T11:32:42.994801+0000","last_active":"2024-03-24T11:32:53.941090+0000","last_peered":"2024-03-24T11:32:53.941090+0000","last_clean":"2024-03-24T11:32:53.941090+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:32:53.941090+0000","last_undegraded":"2024-03-24T11:32:53.941090+0000","last_fullsized":"2024-03-24T11:32:53.941090+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.994694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.994694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.156201+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":381,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007253+0000","last_change":"2024-03-24T11:31:08.903891+0000","last_active":"2024-03-24T11:32:59.007253+0000","last_peered":"2024-03-24T11:32:59.007253+0000","last_clean":"2024-03-24T11:32:59.007253+0000","last_became_active":"2024-03-24T11:30:35.356166+0000","last_became_peered":"2024-03-24T11:30:35.356166+0000","last_unstale":"2024-03-24T11:32:59.007253+0000","last_undegraded":"2024-03-24T11:32:59.007253+0000","last_fullsized":"2024-03-24T11:32:59.007253+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:08.903763+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.903763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940578+0000","last_change":"2024-03-24T11:32:52.326526+0000","last_active":"2024-03-24T11:32:53.940578+0000","last_peered":"2024-03-24T11:32:53.940578+0000","last_clean":"2024-03-24T11:32:53.940578+0000","last_became_active":"2024-03-24T11:32:52.326215+0000","last_became_peered":"2024-03-24T11:32:52.326215+0000","last_unstale":"2024-03-24T11:32:53.940578+0000","last_undegraded":"2024-03-24T11:32:53.940578+0000","last_fullsized":"2024-03-24T11:32:53.940578+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.170611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":465,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007365+0000","last_change":"2024-03-24T11:31:28.613941+0000","last_active":"2024-03-24T11:32:59.007365+0000","last_peered":"2024-03-24T11:32:59.007365+0000","last_clean":"2024-03-24T11:32:59.007365+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:32:59.007365+0000","last_undegraded":"2024-03-24T11:32:59.007365+0000","last_fullsized":"2024-03-24T11:32:59.007365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.613752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.613752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"213'40","reported_seq":721,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.008217+0000","last_change":"2024-03-24T11:32:43.782374+0000","last_active":"2024-03-24T11:32:59.008217+0000","last_peered":"2024-03-24T11:32:59.008217+0000","last_clean":"2024-03-24T11:32:59.008217+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:32:59.008217+0000","last_undegraded":"2024-03-24T11:32:59.008217+0000","last_fullsized":"2024-03-24T11:32:59.008217+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'40","last_scrub_stamp":"2024-03-24T11:32:43.782256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.782256+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:01.406173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":471,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328909+0000","last_change":"2024-03-24T11:31:05.775644+0000","last_active":"2024-03-24T11:32:53.328909+0000","last_peered":"2024-03-24T11:32:53.328909+0000","last_clean":"2024-03-24T11:32:53.328909+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:32:53.328909+0000","last_undegraded":"2024-03-24T11:32:53.328909+0000","last_fullsized":"2024-03-24T11:32:53.328909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:05.775534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:05.775534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":492,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464905+0000","last_change":"2024-03-24T11:32:28.707278+0000","last_active":"2024-03-24T11:32:58.464905+0000","last_peered":"2024-03-24T11:32:58.464905+0000","last_clean":"2024-03-24T11:32:58.464905+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:32:58.464905+0000","last_undegraded":"2024-03-24T11:32:58.464905+0000","last_fullsized":"2024-03-24T11:32:58.464905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:32:28.707184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.707184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:33.691220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":445,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347255+0000","last_change":"2024-03-24T11:29:18.915411+0000","last_active":"2024-03-24T11:32:58.347255+0000","last_peered":"2024-03-24T11:32:58.347255+0000","last_clean":"2024-03-24T11:32:58.347255+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:32:58.347255+0000","last_undegraded":"2024-03-24T11:32:58.347255+0000","last_fullsized":"2024-03-24T11:32:58.347255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:18.915289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:18.915289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":628,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464742+0000","last_change":"2024-03-24T11:32:29.691475+0000","last_active":"2024-03-24T11:32:58.464742+0000","last_peered":"2024-03-24T11:32:58.464742+0000","last_clean":"2024-03-24T11:32:58.464742+0000","last_became_active":"2024-03-24T11:28:38.173832+0000","last_became_peered":"2024-03-24T11:28:38.173832+0000","last_unstale":"2024-03-24T11:32:58.464742+0000","last_undegraded":"2024-03-24T11:32:58.464742+0000","last_fullsized":"2024-03-24T11:32:58.464742+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:29.691383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:29.691383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.888681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347477+0000","last_change":"2024-03-24T11:32:52.671933+0000","last_active":"2024-03-24T11:32:58.347477+0000","last_peered":"2024-03-24T11:32:58.347477+0000","last_clean":"2024-03-24T11:32:58.347477+0000","last_became_active":"2024-03-24T11:32:52.671650+0000","last_became_peered":"2024-03-24T11:32:52.671650+0000","last_unstale":"2024-03-24T11:32:58.347477+0000","last_undegraded":"2024-03-24T11:32:58.347477+0000","last_fullsized":"2024-03-24T11:32:58.347477+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.569702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"213'15","reported_seq":502,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007864+0000","last_change":"2024-03-24T11:32:37.060211+0000","last_active":"2024-03-24T11:32:59.007864+0000","last_peered":"2024-03-24T11:32:59.007864+0000","last_clean":"2024-03-24T11:32:59.007864+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:32:59.007864+0000","last_undegraded":"2024-03-24T11:32:59.007864+0000","last_fullsized":"2024-03-24T11:32:59.007864+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:32:37.060067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.060067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.865083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":487,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465206+0000","last_change":"2024-03-24T11:31:20.465180+0000","last_active":"2024-03-24T11:32:58.465206+0000","last_peered":"2024-03-24T11:32:58.465206+0000","last_clean":"2024-03-24T11:32:58.465206+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:32:58.465206+0000","last_undegraded":"2024-03-24T11:32:58.465206+0000","last_fullsized":"2024-03-24T11:32:58.465206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:20.465096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:20.465096+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614887+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":481,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347866+0000","last_change":"2024-03-24T11:32:27.943222+0000","last_active":"2024-03-24T11:32:58.347866+0000","last_peered":"2024-03-24T11:32:58.347866+0000","last_clean":"2024-03-24T11:32:58.347866+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:32:58.347866+0000","last_undegraded":"2024-03-24T11:32:58.347866+0000","last_fullsized":"2024-03-24T11:32:58.347866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.943114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.943114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.987411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":461,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007259+0000","last_change":"2024-03-24T11:31:23.762948+0000","last_active":"2024-03-24T11:32:59.007259+0000","last_peered":"2024-03-24T11:32:59.007259+0000","last_clean":"2024-03-24T11:32:59.007259+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:32:59.007259+0000","last_undegraded":"2024-03-24T11:32:59.007259+0000","last_fullsized":"2024-03-24T11:32:59.007259+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:23.762486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:23.762486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070548+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":456,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329443+0000","last_change":"2024-03-24T11:31:33.687541+0000","last_active":"2024-03-24T11:32:53.329443+0000","last_peered":"2024-03-24T11:32:53.329443+0000","last_clean":"2024-03-24T11:32:53.329443+0000","last_became_active":"2024-03-24T11:28:43.192246+0000","last_became_peered":"2024-03-24T11:28:43.192246+0000","last_unstale":"2024-03-24T11:32:53.329443+0000","last_undegraded":"2024-03-24T11:32:53.329443+0000","last_fullsized":"2024-03-24T11:32:53.329443+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:33.687261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.687261+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007339+0000","last_change":"2024-03-24T11:32:52.673412+0000","last_active":"2024-03-24T11:32:59.007339+0000","last_peered":"2024-03-24T11:32:59.007339+0000","last_clean":"2024-03-24T11:32:59.007339+0000","last_became_active":"2024-03-24T11:32:52.673121+0000","last_became_peered":"2024-03-24T11:32:52.673121+0000","last_unstale":"2024-03-24T11:32:59.007339+0000","last_undegraded":"2024-03-24T11:32:59.007339+0000","last_fullsized":"2024-03-24T11:32:59.007339+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:58.978249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"213'44","reported_seq":547,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940671+0000","last_change":"2024-03-24T11:31:21.903188+0000","last_active":"2024-03-24T11:32:53.940671+0000","last_peered":"2024-03-24T11:32:53.940671+0000","last_clean":"2024-03-24T11:32:53.940671+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:32:53.940671+0000","last_undegraded":"2024-03-24T11:32:53.940671+0000","last_fullsized":"2024-03-24T11:32:53.940671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:31:21.903023+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:31:21.903023+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.939534+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":85,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465418+0000","last_change":"2024-03-24T11:32:38.827248+0000","last_active":"2024-03-24T11:32:58.465418+0000","last_peered":"2024-03-24T11:32:58.465418+0000","last_clean":"2024-03-24T11:32:58.465418+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:32:58.465418+0000","last_undegraded":"2024-03-24T11:32:58.465418+0000","last_fullsized":"2024-03-24T11:32:58.465418+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.827167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.827167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.576619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":481,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.348333+0000","last_change":"2024-03-24T11:31:22.849153+0000","last_active":"2024-03-24T11:32:58.348333+0000","last_peered":"2024-03-24T11:32:58.348333+0000","last_clean":"2024-03-24T11:32:58.348333+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:32:58.348333+0000","last_undegraded":"2024-03-24T11:32:58.348333+0000","last_fullsized":"2024-03-24T11:32:58.348333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.849049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.849049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":85,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346993+0000","last_change":"2024-03-24T11:31:32.996560+0000","last_active":"2024-03-24T11:32:58.346993+0000","last_peered":"2024-03-24T11:32:58.346993+0000","last_clean":"2024-03-24T11:32:58.346993+0000","last_became_active":"2024-03-24T11:30:35.355210+0000","last_became_peered":"2024-03-24T11:30:35.355210+0000","last_unstale":"2024-03-24T11:32:58.346993+0000","last_undegraded":"2024-03-24T11:32:58.346993+0000","last_fullsized":"2024-03-24T11:32:58.346993+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:32.996416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:32.996416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":638,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941300+0000","last_change":"2024-03-24T11:32:34.973136+0000","last_active":"2024-03-24T11:32:53.941300+0000","last_peered":"2024-03-24T11:32:53.941300+0000","last_clean":"2024-03-24T11:32:53.941300+0000","last_became_active":"2024-03-24T11:28:43.191874+0000","last_became_peered":"2024-03-24T11:28:43.191874+0000","last_unstale":"2024-03-24T11:32:53.941300+0000","last_undegraded":"2024-03-24T11:32:53.941300+0000","last_fullsized":"2024-03-24T11:32:53.941300+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.973008+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.990059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":13,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941192+0000","last_change":"2024-03-24T11:32:52.673866+0000","last_active":"2024-03-24T11:32:53.941192+0000","last_peered":"2024-03-24T11:32:53.941192+0000","last_clean":"2024-03-24T11:32:53.941192+0000","last_became_active":"2024-03-24T11:32:52.673498+0000","last_became_peered":"2024-03-24T11:32:52.673498+0000","last_unstale":"2024-03-24T11:32:53.941192+0000","last_undegraded":"2024-03-24T11:32:53.941192+0000","last_fullsized":"2024-03-24T11:32:53.941192+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:16.878452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":489,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346755+0000","last_change":"2024-03-24T11:32:44.207912+0000","last_active":"2024-03-24T11:32:58.346755+0000","last_peered":"2024-03-24T11:32:58.346755+0000","last_clean":"2024-03-24T11:32:58.346755+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:32:58.346755+0000","last_undegraded":"2024-03-24T11:32:58.346755+0000","last_fullsized":"2024-03-24T11:32:58.346755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:44.207817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:44.207817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.132453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"213'18","reported_seq":499,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328997+0000","last_change":"2024-03-24T11:31:11.798229+0000","last_active":"2024-03-24T11:32:53.328997+0000","last_peered":"2024-03-24T11:32:53.328997+0000","last_clean":"2024-03-24T11:32:53.328997+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:32:53.328997+0000","last_undegraded":"2024-03-24T11:32:53.328997+0000","last_fullsized":"2024-03-24T11:32:53.328997+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.798089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.798089+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":75,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329617+0000","last_change":"2024-03-24T11:30:35.363477+0000","last_active":"2024-03-24T11:32:53.329617+0000","last_peered":"2024-03-24T11:32:53.329617+0000","last_clean":"2024-03-24T11:32:53.329617+0000","last_became_active":"2024-03-24T11:30:35.356055+0000","last_became_peered":"2024-03-24T11:30:35.356055+0000","last_unstale":"2024-03-24T11:32:53.329617+0000","last_undegraded":"2024-03-24T11:32:53.329617+0000","last_fullsized":"2024-03-24T11:32:53.329617+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:07.709842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:30:07.709842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":445,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346228+0000","last_change":"2024-03-24T11:30:12.206577+0000","last_active":"2024-03-24T11:32:58.346228+0000","last_peered":"2024-03-24T11:32:58.346228+0000","last_clean":"2024-03-24T11:32:58.346228+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:32:58.346228+0000","last_undegraded":"2024-03-24T11:32:58.346228+0000","last_fullsized":"2024-03-24T11:32:58.346228+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:12.206287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:12.206287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":616,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345993+0000","last_change":"2024-03-24T11:29:10.818100+0000","last_active":"2024-03-24T11:32:58.345993+0000","last_peered":"2024-03-24T11:32:58.345993+0000","last_clean":"2024-03-24T11:32:58.345993+0000","last_became_active":"2024-03-24T11:28:43.189494+0000","last_became_peered":"2024-03-24T11:28:43.189494+0000","last_unstale":"2024-03-24T11:32:58.345993+0000","last_undegraded":"2024-03-24T11:32:58.345993+0000","last_fullsized":"2024-03-24T11:32:58.345993+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:29:10.817950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:29:10.817950+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":15,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465292+0000","last_change":"2024-03-24T11:32:52.327253+0000","last_active":"2024-03-24T11:32:58.465292+0000","last_peered":"2024-03-24T11:32:58.465292+0000","last_clean":"2024-03-24T11:32:58.465292+0000","last_became_active":"2024-03-24T11:32:52.326946+0000","last_became_peered":"2024-03-24T11:32:52.326946+0000","last_unstale":"2024-03-24T11:32:58.465292+0000","last_undegraded":"2024-03-24T11:32:58.465292+0000","last_fullsized":"2024-03-24T11:32:58.465292+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:11.213465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"238'9","reported_seq":145,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940513+0000","last_change":"2024-03-24T11:32:43.964046+0000","last_active":"2024-03-24T11:32:53.940513+0000","last_peered":"2024-03-24T11:32:53.940513+0000","last_clean":"2024-03-24T11:32:53.940513+0000","last_became_active":"2024-03-24T11:30:35.359635+0000","last_became_peered":"2024-03-24T11:30:35.359635+0000","last_unstale":"2024-03-24T11:32:53.940513+0000","last_undegraded":"2024-03-24T11:32:53.940513+0000","last_fullsized":"2024-03-24T11:32:53.940513+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'9","last_scrub_stamp":"2024-03-24T11:32:43.963946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.963946+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.444511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"213'34","reported_seq":527,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464561+0000","last_change":"2024-03-24T11:31:12.435804+0000","last_active":"2024-03-24T11:32:58.464561+0000","last_peered":"2024-03-24T11:32:58.464561+0000","last_clean":"2024-03-24T11:32:58.464561+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:32:58.464561+0000","last_undegraded":"2024-03-24T11:32:58.464561+0000","last_fullsized":"2024-03-24T11:32:58.464561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:31:12.435639+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:31:12.435639+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614869+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":277,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329653+0000","last_change":"2024-03-24T11:30:35.356491+0000","last_active":"2024-03-24T11:32:53.329653+0000","last_peered":"2024-03-24T11:32:53.329653+0000","last_clean":"2024-03-24T11:32:53.329653+0000","last_became_active":"2024-03-24T11:30:35.356100+0000","last_became_peered":"2024-03-24T11:30:35.356100+0000","last_unstale":"2024-03-24T11:32:53.329653+0000","last_undegraded":"2024-03-24T11:32:53.329653+0000","last_fullsized":"2024-03-24T11:32:53.329653+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:30:11.625473+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.625473+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":469,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465045+0000","last_change":"2024-03-24T11:31:14.458586+0000","last_active":"2024-03-24T11:32:58.465045+0000","last_peered":"2024-03-24T11:32:58.465045+0000","last_clean":"2024-03-24T11:32:58.465045+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:32:58.465045+0000","last_undegraded":"2024-03-24T11:32:58.465045+0000","last_fullsized":"2024-03-24T11:32:58.465045+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.458470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.458470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614849+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":541,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329752+0000","last_change":"2024-03-24T11:31:04.735517+0000","last_active":"2024-03-24T11:32:53.329752+0000","last_peered":"2024-03-24T11:32:53.329752+0000","last_clean":"2024-03-24T11:32:53.329752+0000","last_became_active":"2024-03-24T11:30:35.354637+0000","last_became_peered":"2024-03-24T11:30:35.354637+0000","last_unstale":"2024-03-24T11:32:53.329752+0000","last_undegraded":"2024-03-24T11:32:53.329752+0000","last_fullsized":"2024-03-24T11:32:53.329752+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:04.735433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:04.735433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"213'15","reported_seq":473,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347690+0000","last_change":"2024-03-24T11:30:14.150582+0000","last_active":"2024-03-24T11:32:58.347690+0000","last_peered":"2024-03-24T11:32:58.347690+0000","last_clean":"2024-03-24T11:32:58.347690+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:32:58.347690+0000","last_undegraded":"2024-03-24T11:32:58.347690+0000","last_fullsized":"2024-03-24T11:32:58.347690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:14.150454+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.150454+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":495,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940483+0000","last_change":"2024-03-24T11:32:40.958471+0000","last_active":"2024-03-24T11:32:53.940483+0000","last_peered":"2024-03-24T11:32:53.940483+0000","last_clean":"2024-03-24T11:32:53.940483+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:32:53.940483+0000","last_undegraded":"2024-03-24T11:32:53.940483+0000","last_fullsized":"2024-03-24T11:32:53.940483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:40.958389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.958389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.190213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":472,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465334+0000","last_change":"2024-03-24T11:32:39.835187+0000","last_active":"2024-03-24T11:32:58.465334+0000","last_peered":"2024-03-24T11:32:58.465334+0000","last_clean":"2024-03-24T11:32:58.465334+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:32:58.465334+0000","last_undegraded":"2024-03-24T11:32:58.465334+0000","last_fullsized":"2024-03-24T11:32:58.465334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.835097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.835097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:03.202024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":90,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941146+0000","last_change":"2024-03-24T11:32:32.920081+0000","last_active":"2024-03-24T11:32:53.941146+0000","last_peered":"2024-03-24T11:32:53.941146+0000","last_clean":"2024-03-24T11:32:53.941146+0000","last_became_active":"2024-03-24T11:30:35.353212+0000","last_became_peered":"2024-03-24T11:30:35.353212+0000","last_unstale":"2024-03-24T11:32:53.941146+0000","last_undegraded":"2024-03-24T11:32:53.941146+0000","last_fullsized":"2024-03-24T11:32:53.941146+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.919963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:34.622437+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464674+0000","last_change":"2024-03-24T11:31:24.533967+0000","last_active":"2024-03-24T11:32:58.464674+0000","last_peered":"2024-03-24T11:32:58.464674+0000","last_clean":"2024-03-24T11:32:58.464674+0000","last_became_active":"2024-03-24T11:28:43.189410+0000","last_became_peered":"2024-03-24T11:28:43.189410+0000","last_unstale":"2024-03-24T11:32:58.464674+0000","last_undegraded":"2024-03-24T11:32:58.464674+0000","last_fullsized":"2024-03-24T11:32:58.464674+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_clean_scrub_stamp":"2024-03-24T11:31:24.533874+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614923+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":634,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006957+0000","last_change":"2024-03-24T11:32:33.790867+0000","last_active":"2024-03-24T11:32:59.006957+0000","last_peered":"2024-03-24T11:32:59.006957+0000","last_clean":"2024-03-24T11:32:59.006957+0000","last_became_active":"2024-03-24T11:28:43.192449+0000","last_became_peered":"2024-03-24T11:28:43.192449+0000","last_unstale":"2024-03-24T11:32:59.006957+0000","last_undegraded":"2024-03-24T11:32:59.006957+0000","last_fullsized":"2024-03-24T11:32:59.006957+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.790777+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.025287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"213'19","reported_seq":505,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007892+0000","last_change":"2024-03-24T11:32:32.023601+0000","last_active":"2024-03-24T11:32:59.007892+0000","last_peered":"2024-03-24T11:32:59.007892+0000","last_clean":"2024-03-24T11:32:59.007892+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:32:59.007892+0000","last_undegraded":"2024-03-24T11:32:59.007892+0000","last_fullsized":"2024-03-24T11:32:59.007892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:32.023502+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.023502+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.100674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":75,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328973+0000","last_change":"2024-03-24T11:30:35.359322+0000","last_active":"2024-03-24T11:32:53.328973+0000","last_peered":"2024-03-24T11:32:53.328973+0000","last_clean":"2024-03-24T11:32:53.328973+0000","last_became_active":"2024-03-24T11:30:35.358928+0000","last_became_peered":"2024-03-24T11:30:35.358928+0000","last_unstale":"2024-03-24T11:32:53.328973+0000","last_undegraded":"2024-03-24T11:32:53.328973+0000","last_fullsized":"2024-03-24T11:32:53.328973+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:21.749544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:21.749544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.760837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":448,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347682+0000","last_change":"2024-03-24T11:29:26.014447+0000","last_active":"2024-03-24T11:32:58.347682+0000","last_peered":"2024-03-24T11:32:58.347682+0000","last_clean":"2024-03-24T11:32:58.347682+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:32:58.347682+0000","last_undegraded":"2024-03-24T11:32:58.347682+0000","last_fullsized":"2024-03-24T11:32:58.347682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:26.014199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:29:26.014199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":476,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.348496+0000","last_change":"2024-03-24T11:32:26.898659+0000","last_active":"2024-03-24T11:32:58.348496+0000","last_peered":"2024-03-24T11:32:58.348496+0000","last_clean":"2024-03-24T11:32:58.348496+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:32:58.348496+0000","last_undegraded":"2024-03-24T11:32:58.348496+0000","last_fullsized":"2024-03-24T11:32:58.348496+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:26.898583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.898583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.612791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"213'15","reported_seq":501,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465483+0000","last_change":"2024-03-24T11:31:27.541622+0000","last_active":"2024-03-24T11:32:58.465483+0000","last_peered":"2024-03-24T11:32:58.465483+0000","last_clean":"2024-03-24T11:32:58.465483+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:32:58.465483+0000","last_undegraded":"2024-03-24T11:32:58.465483+0000","last_fullsized":"2024-03-24T11:32:58.465483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:27.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.541510+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614942+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":481,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007260+0000","last_change":"2024-03-24T11:32:45.985820+0000","last_active":"2024-03-24T11:32:59.007260+0000","last_peered":"2024-03-24T11:32:59.007260+0000","last_clean":"2024-03-24T11:32:59.007260+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:32:59.007260+0000","last_undegraded":"2024-03-24T11:32:59.007260+0000","last_fullsized":"2024-03-24T11:32:59.007260+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.985741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.985741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:14.491398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":634,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346458+0000","last_change":"2024-03-24T11:31:13.201069+0000","last_active":"2024-03-24T11:32:58.346458+0000","last_peered":"2024-03-24T11:32:58.346458+0000","last_clean":"2024-03-24T11:32:58.346458+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:32:58.346458+0000","last_undegraded":"2024-03-24T11:32:58.346458+0000","last_fullsized":"2024-03-24T11:32:58.346458+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:13.200503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.200503+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":458,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329770+0000","last_change":"2024-03-24T11:32:27.764304+0000","last_active":"2024-03-24T11:32:53.329770+0000","last_peered":"2024-03-24T11:32:53.329770+0000","last_clean":"2024-03-24T11:32:53.329770+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:32:53.329770+0000","last_undegraded":"2024-03-24T11:32:53.329770+0000","last_fullsized":"2024-03-24T11:32:53.329770+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.764203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.764203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.837847+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464732+0000","last_change":"2024-03-24T11:31:25.526652+0000","last_active":"2024-03-24T11:32:58.464732+0000","last_peered":"2024-03-24T11:32:58.464732+0000","last_clean":"2024-03-24T11:32:58.464732+0000","last_became_active":"2024-03-24T11:28:43.188941+0000","last_became_peered":"2024-03-24T11:28:43.188941+0000","last_unstale":"2024-03-24T11:32:58.464732+0000","last_undegraded":"2024-03-24T11:32:58.464732+0000","last_fullsized":"2024-03-24T11:32:58.464732+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.526522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614800+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"213'20","reported_seq":483,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346384+0000","last_change":"2024-03-24T11:29:32.055362+0000","last_active":"2024-03-24T11:32:58.346384+0000","last_peered":"2024-03-24T11:32:58.346384+0000","last_clean":"2024-03-24T11:32:58.346384+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:32:58.346384+0000","last_undegraded":"2024-03-24T11:32:58.346384+0000","last_fullsized":"2024-03-24T11:32:58.346384+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:29:32.055233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:29:32.055233+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":83,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329041+0000","last_change":"2024-03-24T11:31:30.728130+0000","last_active":"2024-03-24T11:32:53.329041+0000","last_peered":"2024-03-24T11:32:53.329041+0000","last_clean":"2024-03-24T11:32:53.329041+0000","last_became_active":"2024-03-24T11:30:35.353946+0000","last_became_peered":"2024-03-24T11:30:35.353946+0000","last_unstale":"2024-03-24T11:32:53.329041+0000","last_undegraded":"2024-03-24T11:32:53.329041+0000","last_fullsized":"2024-03-24T11:32:53.329041+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:30.728004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.728004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.761082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":649,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465285+0000","last_change":"2024-03-24T11:31:36.682644+0000","last_active":"2024-03-24T11:32:58.465285+0000","last_peered":"2024-03-24T11:32:58.465285+0000","last_clean":"2024-03-24T11:32:58.465285+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:32:58.465285+0000","last_undegraded":"2024-03-24T11:32:58.465285+0000","last_fullsized":"2024-03-24T11:32:58.465285+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:36.682542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.682542+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614905+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":460,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007682+0000","last_change":"2024-03-24T11:32:31.071719+0000","last_active":"2024-03-24T11:32:59.007682+0000","last_peered":"2024-03-24T11:32:59.007682+0000","last_clean":"2024-03-24T11:32:59.007682+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:32:59.007682+0000","last_undegraded":"2024-03-24T11:32:59.007682+0000","last_fullsized":"2024-03-24T11:32:59.007682+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:31.071621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:31.071621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.164745+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":472,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007011+0000","last_change":"2024-03-24T11:31:06.835419+0000","last_active":"2024-03-24T11:32:59.007011+0000","last_peered":"2024-03-24T11:32:59.007011+0000","last_clean":"2024-03-24T11:32:59.007011+0000","last_became_active":"2024-03-24T11:28:43.188012+0000","last_became_peered":"2024-03-24T11:28:43.188012+0000","last_unstale":"2024-03-24T11:32:59.007011+0000","last_undegraded":"2024-03-24T11:32:59.007011+0000","last_fullsized":"2024-03-24T11:32:59.007011+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:06.835322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.835322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070501+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"213'24","reported_seq":511,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329098+0000","last_change":"2024-03-24T11:32:43.646863+0000","last_active":"2024-03-24T11:32:53.329098+0000","last_peered":"2024-03-24T11:32:53.329098+0000","last_clean":"2024-03-24T11:32:53.329098+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:32:53.329098+0000","last_undegraded":"2024-03-24T11:32:53.329098+0000","last_fullsized":"2024-03-24T11:32:53.329098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:43.646773+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.646773+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.831542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":471,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329301+0000","last_change":"2024-03-24T11:31:37.636078+0000","last_active":"2024-03-24T11:32:53.329301+0000","last_peered":"2024-03-24T11:32:53.329301+0000","last_clean":"2024-03-24T11:32:53.329301+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:32:53.329301+0000","last_undegraded":"2024-03-24T11:32:53.329301+0000","last_fullsized":"2024-03-24T11:32:53.329301+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:37.635705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.635705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:58.761067+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":489,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007251+0000","last_change":"2024-03-24T11:32:36.752484+0000","last_active":"2024-03-24T11:32:59.007251+0000","last_peered":"2024-03-24T11:32:59.007251+0000","last_clean":"2024-03-24T11:32:59.007251+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:32:59.007251+0000","last_undegraded":"2024-03-24T11:32:59.007251+0000","last_fullsized":"2024-03-24T11:32:59.007251+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.752406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.752406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:47.500544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":453,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345787+0000","last_change":"2024-03-24T11:32:32.077273+0000","last_active":"2024-03-24T11:32:58.345787+0000","last_peered":"2024-03-24T11:32:58.345787+0000","last_clean":"2024-03-24T11:32:58.345787+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:32:58.345787+0000","last_undegraded":"2024-03-24T11:32:58.345787+0000","last_fullsized":"2024-03-24T11:32:58.345787+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.077150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.077150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:35.573524+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"213'15","reported_seq":502,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007461+0000","last_change":"2024-03-24T11:31:36.651472+0000","last_active":"2024-03-24T11:32:59.007461+0000","last_peered":"2024-03-24T11:32:59.007461+0000","last_clean":"2024-03-24T11:32:59.007461+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:32:59.007461+0000","last_undegraded":"2024-03-24T11:32:59.007461+0000","last_fullsized":"2024-03-24T11:32:59.007461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:36.651340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.651340+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.987823+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":84,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346623+0000","last_change":"2024-03-24T11:32:45.875224+0000","last_active":"2024-03-24T11:32:58.346623+0000","last_peered":"2024-03-24T11:32:58.346623+0000","last_clean":"2024-03-24T11:32:58.346623+0000","last_became_active":"2024-03-24T11:30:35.353230+0000","last_became_peered":"2024-03-24T11:30:35.353230+0000","last_unstale":"2024-03-24T11:32:58.346623+0000","last_undegraded":"2024-03-24T11:32:58.346623+0000","last_fullsized":"2024-03-24T11:32:58.346623+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.875145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.875145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:14.877519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":448,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346126+0000","last_change":"2024-03-24T11:29:29.048318+0000","last_active":"2024-03-24T11:32:58.346126+0000","last_peered":"2024-03-24T11:32:58.346126+0000","last_clean":"2024-03-24T11:32:58.346126+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:32:58.346126+0000","last_undegraded":"2024-03-24T11:32:58.346126+0000","last_fullsized":"2024-03-24T11:32:58.346126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:29.046315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:29:29.046315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":461,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006587+0000","last_change":"2024-03-24T11:32:24.051941+0000","last_active":"2024-03-24T11:32:59.006587+0000","last_peered":"2024-03-24T11:32:59.006587+0000","last_clean":"2024-03-24T11:32:59.006587+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:32:59.006587+0000","last_undegraded":"2024-03-24T11:32:59.006587+0000","last_fullsized":"2024-03-24T11:32:59.006587+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.051777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.051777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.050306+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":621,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347783+0000","last_change":"2024-03-24T11:32:40.219356+0000","last_active":"2024-03-24T11:32:58.347783+0000","last_peered":"2024-03-24T11:32:58.347783+0000","last_clean":"2024-03-24T11:32:58.347783+0000","last_became_active":"2024-03-24T11:28:43.192919+0000","last_became_peered":"2024-03-24T11:28:43.192919+0000","last_unstale":"2024-03-24T11:32:58.347783+0000","last_undegraded":"2024-03-24T11:32:58.347783+0000","last_fullsized":"2024-03-24T11:32:58.347783+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:40.219272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.219272+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.153521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"213'19","reported_seq":484,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345934+0000","last_change":"2024-03-24T11:30:13.139116+0000","last_active":"2024-03-24T11:32:58.345934+0000","last_peered":"2024-03-24T11:32:58.345934+0000","last_clean":"2024-03-24T11:32:58.345934+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:32:58.345934+0000","last_undegraded":"2024-03-24T11:32:58.345934+0000","last_fullsized":"2024-03-24T11:32:58.345934+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:30:13.138980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:13.138980+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":480,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465243+0000","last_change":"2024-03-24T11:31:40.668063+0000","last_active":"2024-03-24T11:32:58.465243+0000","last_peered":"2024-03-24T11:32:58.465243+0000","last_clean":"2024-03-24T11:32:58.465243+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:32:58.465243+0000","last_undegraded":"2024-03-24T11:32:58.465243+0000","last_fullsized":"2024-03-24T11:32:58.465243+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:40.667849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:40.667849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:09.047430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":83,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328568+0000","last_change":"2024-03-24T11:32:23.732096+0000","last_active":"2024-03-24T11:32:53.328568+0000","last_peered":"2024-03-24T11:32:53.328568+0000","last_clean":"2024-03-24T11:32:53.328568+0000","last_became_active":"2024-03-24T11:30:35.353571+0000","last_became_peered":"2024-03-24T11:30:35.353571+0000","last_unstale":"2024-03-24T11:32:53.328568+0000","last_undegraded":"2024-03-24T11:32:53.328568+0000","last_fullsized":"2024-03-24T11:32:53.328568+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:23.731967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.731967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:38.504397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":452,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346333+0000","last_change":"2024-03-24T11:30:24.313618+0000","last_active":"2024-03-24T11:32:58.346333+0000","last_peered":"2024-03-24T11:32:58.346333+0000","last_clean":"2024-03-24T11:32:58.346333+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:32:58.346333+0000","last_undegraded":"2024-03-24T11:32:58.346333+0000","last_fullsized":"2024-03-24T11:32:58.346333+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:24.313152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.313152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"213'10","reported_seq":467,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345489+0000","last_change":"2024-03-24T11:29:30.032558+0000","last_active":"2024-03-24T11:32:58.345489+0000","last_peered":"2024-03-24T11:32:58.345489+0000","last_clean":"2024-03-24T11:32:58.345489+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:32:58.345489+0000","last_undegraded":"2024-03-24T11:32:58.345489+0000","last_fullsized":"2024-03-24T11:32:58.345489+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:29:30.032476+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:29:30.032476+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":480,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007910+0000","last_change":"2024-03-24T11:32:36.013375+0000","last_active":"2024-03-24T11:32:59.007910+0000","last_peered":"2024-03-24T11:32:59.007910+0000","last_clean":"2024-03-24T11:32:59.007910+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:32:59.007910+0000","last_undegraded":"2024-03-24T11:32:59.007910+0000","last_fullsized":"2024-03-24T11:32:59.007910+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.013287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.013287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:44.208803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":487,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.940645+0000","last_change":"2024-03-24T11:32:38.991091+0000","last_active":"2024-03-24T11:32:53.940645+0000","last_peered":"2024-03-24T11:32:53.940645+0000","last_clean":"2024-03-24T11:32:53.940645+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:32:53.940645+0000","last_undegraded":"2024-03-24T11:32:53.940645+0000","last_fullsized":"2024-03-24T11:32:53.940645+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.991009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.991009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:05.887138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":93,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.348145+0000","last_change":"2024-03-24T11:32:43.213245+0000","last_active":"2024-03-24T11:32:58.348145+0000","last_peered":"2024-03-24T11:32:58.348145+0000","last_clean":"2024-03-24T11:32:58.348145+0000","last_became_active":"2024-03-24T11:30:35.353704+0000","last_became_peered":"2024-03-24T11:30:35.353704+0000","last_unstale":"2024-03-24T11:32:58.348145+0000","last_undegraded":"2024-03-24T11:32:58.348145+0000","last_fullsized":"2024-03-24T11:32:58.348145+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:43.213130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.213130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:06.338202+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"213'24","reported_seq":527,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941409+0000","last_change":"2024-03-24T11:32:30.897616+0000","last_active":"2024-03-24T11:32:53.941409+0000","last_peered":"2024-03-24T11:32:53.941409+0000","last_clean":"2024-03-24T11:32:53.941409+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:32:53.941409+0000","last_undegraded":"2024-03-24T11:32:53.941409+0000","last_fullsized":"2024-03-24T11:32:53.941409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:30.897405+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.897405+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.068479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":464,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345464+0000","last_change":"2024-03-24T11:32:30.073961+0000","last_active":"2024-03-24T11:32:58.345464+0000","last_peered":"2024-03-24T11:32:58.345464+0000","last_clean":"2024-03-24T11:32:58.345464+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:32:58.345464+0000","last_undegraded":"2024-03-24T11:32:58.345464+0000","last_fullsized":"2024-03-24T11:32:58.345464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:30.073873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.073873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:52.383142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":466,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.008036+0000","last_change":"2024-03-24T11:31:14.751432+0000","last_active":"2024-03-24T11:32:59.008036+0000","last_peered":"2024-03-24T11:32:59.008036+0000","last_clean":"2024-03-24T11:32:59.008036+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:32:59.008036+0000","last_undegraded":"2024-03-24T11:32:59.008036+0000","last_fullsized":"2024-03-24T11:32:59.008036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.751309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.751309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070532+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":476,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.008416+0000","last_change":"2024-03-24T11:31:41.853862+0000","last_active":"2024-03-24T11:32:59.008416+0000","last_peered":"2024-03-24T11:32:59.008416+0000","last_clean":"2024-03-24T11:32:59.008416+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:32:59.008416+0000","last_undegraded":"2024-03-24T11:32:59.008416+0000","last_fullsized":"2024-03-24T11:32:59.008416+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.853751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:04.343556+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"213'20","reported_seq":508,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007909+0000","last_change":"2024-03-24T11:32:25.904952+0000","last_active":"2024-03-24T11:32:59.007909+0000","last_peered":"2024-03-24T11:32:59.007909+0000","last_clean":"2024-03-24T11:32:59.007909+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:32:59.007909+0000","last_undegraded":"2024-03-24T11:32:59.007909+0000","last_fullsized":"2024-03-24T11:32:59.007909+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:25.904851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.904851+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.755461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":489,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006954+0000","last_change":"2024-03-24T11:31:37.672115+0000","last_active":"2024-03-24T11:32:59.006954+0000","last_peered":"2024-03-24T11:32:59.006954+0000","last_clean":"2024-03-24T11:32:59.006954+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:32:59.006954+0000","last_undegraded":"2024-03-24T11:32:59.006954+0000","last_fullsized":"2024-03-24T11:32:59.006954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:37.671989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.671989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":671,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.941304+0000","last_change":"2024-03-24T11:32:41.963960+0000","last_active":"2024-03-24T11:32:53.941304+0000","last_peered":"2024-03-24T11:32:53.941304+0000","last_clean":"2024-03-24T11:32:53.941304+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:32:53.941304+0000","last_undegraded":"2024-03-24T11:32:53.941304+0000","last_fullsized":"2024-03-24T11:32:53.941304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.963825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.606211+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":123,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464989+0000","last_change":"2024-03-24T11:30:35.355569+0000","last_active":"2024-03-24T11:32:58.464989+0000","last_peered":"2024-03-24T11:32:58.464989+0000","last_clean":"2024-03-24T11:32:58.464989+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:32:58.464989+0000","last_undegraded":"2024-03-24T11:32:58.464989+0000","last_fullsized":"2024-03-24T11:32:58.464989+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:30:22.784182+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:30:22.784182+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"213'18","reported_seq":484,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346337+0000","last_change":"2024-03-24T11:31:11.260142+0000","last_active":"2024-03-24T11:32:58.346337+0000","last_peered":"2024-03-24T11:32:58.346337+0000","last_clean":"2024-03-24T11:32:58.346337+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:32:58.346337+0000","last_undegraded":"2024-03-24T11:32:58.346337+0000","last_fullsized":"2024-03-24T11:32:58.346337+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.259991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.259991+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":462,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329245+0000","last_change":"2024-03-24T11:30:11.883846+0000","last_active":"2024-03-24T11:32:53.329245+0000","last_peered":"2024-03-24T11:32:53.329245+0000","last_clean":"2024-03-24T11:32:53.329245+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:32:53.329245+0000","last_undegraded":"2024-03-24T11:32:53.329245+0000","last_fullsized":"2024-03-24T11:32:53.329245+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:11.883599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:11.883599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":470,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329475+0000","last_change":"2024-03-24T11:32:38.742586+0000","last_active":"2024-03-24T11:32:53.329475+0000","last_peered":"2024-03-24T11:32:53.329475+0000","last_clean":"2024-03-24T11:32:53.329475+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:32:53.329475+0000","last_undegraded":"2024-03-24T11:32:53.329475+0000","last_fullsized":"2024-03-24T11:32:53.329475+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.742478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.742478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.063731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":445,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347483+0000","last_change":"2024-03-24T11:30:08.308986+0000","last_active":"2024-03-24T11:32:58.347483+0000","last_peered":"2024-03-24T11:32:58.347483+0000","last_clean":"2024-03-24T11:32:58.347483+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:32:58.347483+0000","last_undegraded":"2024-03-24T11:32:58.347483+0000","last_fullsized":"2024-03-24T11:32:58.347483+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:08.308860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:08.308860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"213'27","reported_seq":527,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.347952+0000","last_change":"2024-03-24T11:32:35.070335+0000","last_active":"2024-03-24T11:32:58.347952+0000","last_peered":"2024-03-24T11:32:58.347952+0000","last_clean":"2024-03-24T11:32:58.347952+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:32:58.347952+0000","last_undegraded":"2024-03-24T11:32:58.347952+0000","last_fullsized":"2024-03-24T11:32:58.347952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:32:35.070232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:35.070232+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.835251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":472,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007749+0000","last_change":"2024-03-24T11:31:10.852618+0000","last_active":"2024-03-24T11:32:59.007749+0000","last_peered":"2024-03-24T11:32:59.007749+0000","last_clean":"2024-03-24T11:32:59.007749+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:32:59.007749+0000","last_undegraded":"2024-03-24T11:32:59.007749+0000","last_fullsized":"2024-03-24T11:32:59.007749+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:10.852490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.852490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070465+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":480,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.008362+0000","last_change":"2024-03-24T11:32:42.808822+0000","last_active":"2024-03-24T11:32:59.008362+0000","last_peered":"2024-03-24T11:32:59.008362+0000","last_clean":"2024-03-24T11:32:59.008362+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:32:59.008362+0000","last_undegraded":"2024-03-24T11:32:59.008362+0000","last_fullsized":"2024-03-24T11:32:59.008362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.808738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.808738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.742514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":459,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329670+0000","last_change":"2024-03-24T11:31:09.866310+0000","last_active":"2024-03-24T11:32:53.329670+0000","last_peered":"2024-03-24T11:32:53.329670+0000","last_clean":"2024-03-24T11:32:53.329670+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:32:53.329670+0000","last_undegraded":"2024-03-24T11:32:53.329670+0000","last_fullsized":"2024-03-24T11:32:53.329670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:09.866232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:31:09.866232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"213'7","reported_seq":479,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328811+0000","last_change":"2024-03-24T11:31:27.652238+0000","last_active":"2024-03-24T11:32:53.328811+0000","last_peered":"2024-03-24T11:32:53.328811+0000","last_clean":"2024-03-24T11:32:53.328811+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:32:53.328811+0000","last_undegraded":"2024-03-24T11:32:53.328811+0000","last_fullsized":"2024-03-24T11:32:53.328811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:31:27.652037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.652037+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":472,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465142+0000","last_change":"2024-03-24T11:30:14.299035+0000","last_active":"2024-03-24T11:32:58.465142+0000","last_peered":"2024-03-24T11:32:58.465142+0000","last_clean":"2024-03-24T11:32:58.465142+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:32:58.465142+0000","last_undegraded":"2024-03-24T11:32:58.465142+0000","last_fullsized":"2024-03-24T11:32:58.465142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:14.298903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.298903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614671+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":82,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329479+0000","last_change":"2024-03-24T11:31:41.715026+0000","last_active":"2024-03-24T11:32:53.329479+0000","last_peered":"2024-03-24T11:32:53.329479+0000","last_clean":"2024-03-24T11:32:53.329479+0000","last_became_active":"2024-03-24T11:30:35.359239+0000","last_became_peered":"2024-03-24T11:30:35.359239+0000","last_unstale":"2024-03-24T11:32:53.329479+0000","last_undegraded":"2024-03-24T11:32:53.329479+0000","last_fullsized":"2024-03-24T11:32:53.329479+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:11.406185+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":460,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464852+0000","last_change":"2024-03-24T11:30:17.465431+0000","last_active":"2024-03-24T11:32:58.464852+0000","last_peered":"2024-03-24T11:32:58.464852+0000","last_clean":"2024-03-24T11:32:58.464852+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:32:58.464852+0000","last_undegraded":"2024-03-24T11:32:58.464852+0000","last_fullsized":"2024-03-24T11:32:58.464852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.465226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.465226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614749+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"213'15","reported_seq":483,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.329070+0000","last_change":"2024-03-24T11:30:15.882584+0000","last_active":"2024-03-24T11:32:53.329070+0000","last_peered":"2024-03-24T11:32:53.329070+0000","last_clean":"2024-03-24T11:32:53.329070+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:32:53.329070+0000","last_undegraded":"2024-03-24T11:32:53.329070+0000","last_fullsized":"2024-03-24T11:32:53.329070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:15.882446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:15.882446+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":480,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.465296+0000","last_change":"2024-03-24T11:31:41.687584+0000","last_active":"2024-03-24T11:32:58.465296+0000","last_peered":"2024-03-24T11:32:58.465296+0000","last_clean":"2024-03-24T11:32:58.465296+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:32:58.465296+0000","last_undegraded":"2024-03-24T11:32:58.465296+0000","last_fullsized":"2024-03-24T11:32:58.465296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.687474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.687474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:06.652087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":472,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.464992+0000","last_change":"2024-03-24T11:31:15.474018+0000","last_active":"2024-03-24T11:32:58.464992+0000","last_peered":"2024-03-24T11:32:58.464992+0000","last_clean":"2024-03-24T11:32:58.464992+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:32:58.464992+0000","last_undegraded":"2024-03-24T11:32:58.464992+0000","last_fullsized":"2024-03-24T11:32:58.464992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.473909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.473909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:02.614781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":460,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006755+0000","last_change":"2024-03-24T11:31:18.759423+0000","last_active":"2024-03-24T11:32:59.006755+0000","last_peered":"2024-03-24T11:32:59.006755+0000","last_clean":"2024-03-24T11:32:59.006755+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:32:59.006755+0000","last_undegraded":"2024-03-24T11:32:59.006755+0000","last_fullsized":"2024-03-24T11:32:59.006755+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:18.759165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.759165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:01.070483+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"213'27","reported_seq":495,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.345936+0000","last_change":"2024-03-24T11:31:41.107651+0000","last_active":"2024-03-24T11:32:58.345936+0000","last_peered":"2024-03-24T11:32:58.345936+0000","last_clean":"2024-03-24T11:32:58.345936+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:32:58.345936+0000","last_undegraded":"2024-03-24T11:32:58.345936+0000","last_fullsized":"2024-03-24T11:32:58.345936+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:31:41.107512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.107512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346824+0000","last_change":"2024-03-24T11:30:23.137546+0000","last_active":"2024-03-24T11:32:58.346824+0000","last_peered":"2024-03-24T11:32:58.346824+0000","last_clean":"2024-03-24T11:32:58.346824+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:32:58.346824+0000","last_undegraded":"2024-03-24T11:32:58.346824+0000","last_fullsized":"2024-03-24T11:32:58.346824+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:23.137393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:23.137393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":462,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328632+0000","last_change":"2024-03-24T11:32:24.692923+0000","last_active":"2024-03-24T11:32:53.328632+0000","last_peered":"2024-03-24T11:32:53.328632+0000","last_clean":"2024-03-24T11:32:53.328632+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:32:53.328632+0000","last_undegraded":"2024-03-24T11:32:53.328632+0000","last_fullsized":"2024-03-24T11:32:53.328632+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.692844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.692844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.656723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":444,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:58.346251+0000","last_change":"2024-03-24T11:31:14.327893+0000","last_active":"2024-03-24T11:32:58.346251+0000","last_peered":"2024-03-24T11:32:58.346251+0000","last_clean":"2024-03-24T11:32:58.346251+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:32:58.346251+0000","last_undegraded":"2024-03-24T11:32:58.346251+0000","last_fullsized":"2024-03-24T11:32:58.346251+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.327777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.327777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"213'15","reported_seq":507,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328538+0000","last_change":"2024-03-24T11:31:08.802687+0000","last_active":"2024-03-24T11:32:53.328538+0000","last_peered":"2024-03-24T11:32:53.328538+0000","last_clean":"2024-03-24T11:32:53.328538+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:32:53.328538+0000","last_undegraded":"2024-03-24T11:32:53.328538+0000","last_fullsized":"2024-03-24T11:32:53.328538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:08.802538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.802538+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":473,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.007713+0000","last_change":"2024-03-24T11:32:35.031834+0000","last_active":"2024-03-24T11:32:59.007713+0000","last_peered":"2024-03-24T11:32:59.007713+0000","last_clean":"2024-03-24T11:32:59.007713+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:32:59.007713+0000","last_undegraded":"2024-03-24T11:32:59.007713+0000","last_fullsized":"2024-03-24T11:32:59.007713+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:35.031735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:35.031735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":480,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-24T11:32:59.006839+0000","last_change":"2024-03-24T11:32:28.712422+0000","last_active":"2024-03-24T11:32:59.006839+0000","last_peered":"2024-03-24T11:32:59.006839+0000","last_clean":"2024-03-24T11:32:59.006839+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:32:59.006839+0000","last_undegraded":"2024-03-24T11:32:59.006839+0000","last_fullsized":"2024-03-24T11:32:59.006839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.712294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.712294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.891340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":458,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-03-24T11:32:53.328807+0000","last_change":"2024-03-24T11:31:07.885306+0000","last_active":"2024-03-24T11:32:53.328807+0000","last_peered":"2024-03-24T11:32:53.328807+0000","last_clean":"2024-03-24T11:32:53.328807+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:32:53.328807+0000","last_undegraded":"2024-03-24T11:32:53.328807+0000","last_fullsized":"2024-03-24T11:32:53.328807+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:07.885177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:07.885177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":150,"num_read_kb":145,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070637,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491480,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253672,"statfs":{"total":95995035648,"available":95491760128,"internally_reserved":0,"allocated":794624,"data_stored":476894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":502462923},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364978,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316876,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93428276,"statfs":{"total":95995035648,"available":95670554624,"internally_reserved":0,"allocated":1110016,"data_stored":804795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575052,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515120,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93230032,"statfs":{"total":95995035648,"available":95467552768,"internally_reserved":0,"allocated":823296,"data_stored":477952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738577,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415464,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329688,"statfs":{"total":95995035648,"available":95569600512,"internally_reserved":0,"allocated":1142784,"data_stored":812450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411527,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334876,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410276,"statfs":{"total":95995035648,"available":95652122624,"internally_reserved":0,"allocated":1056768,"data_stored":739098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149888,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":479612,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93265540,"statfs":{"total":95995035648,"available":95503912960,"internally_reserved":0,"allocated":1425408,"data_stored":1075633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":489683399},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986364,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320744,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93424408,"statfs":{"total":95995035648,"available":95666593792,"internally_reserved":0,"allocated":806912,"data_stored":477322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659319,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491732,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253420,"statfs":{"total":95995035648,"available":95491502080,"internally_reserved":0,"allocated":1052672,"data_stored":739364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:33:02.892 INFO:tasks.thrashosds.thrasher:Setting 4.6 to [5, 2, 3] 2024-03-24T11:33:02.892 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.6', '5', '2', '3'] 2024-03-24T11:33:02.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 4.6 5 2 3 2024-03-24T11:33:02.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:02.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:02.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:03.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:03.043 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:03.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:03.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:03.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:03.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:03.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:03.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:03.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:03.326 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-24T11:33:03.328 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T11:33:03.329 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T11:33:03.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:03.420 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:03.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:03.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[29608]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:33:03.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[29608]: pgmap v596: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:33:03.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[29608]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:33:03.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[37907]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:33:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[37907]: pgmap v596: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:33:03.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[37907]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:33:03.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:03.522 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:03.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:03.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:03.619 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:hints = 1 2024-03-24T11:33:03.619 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T11:33:03.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object prefix: benchmark_data_smithi179_155357 2024-03-24T11:33:03.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:33:03.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-03-24T11:33:03.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:03.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:03 smithi181 ceph-mon[33998]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:33:03.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:03 smithi181 ceph-mon[33998]: pgmap v596: 168 pgs: 168 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:33:03.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:03 smithi181 ceph-mon[33998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:33:03.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:03.686 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:03.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:03.774 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:03.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:03.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:03.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:03.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:04.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:04.027 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:33:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:33:04.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:04.148 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:04.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:04.287 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:04.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:04.424 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:04.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:04.533 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:33:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:33:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:33:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:33:04.533 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:33:04.556 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:04.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:04.619 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 1 16 3420 3404 212.71 212.75 0.00242472 0.00467877 2024-03-24T11:33:04.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:04.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:04.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:04.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:04.844 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:04.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:04.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:04.983 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:04.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:05.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:05.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:05.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:05.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:33:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:33:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:33:05.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:05.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:05.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:05.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T11:33:05.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:05.395 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:05.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:05.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:05.537 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:05.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:05.619 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 2 16 6621 6605 206.378 200.062 0.00237154 0.00483683 2024-03-24T11:33:05.636 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:05 smithi179 ceph-mon[29608]: pgmap v597: 168 pgs: 168 active+clean; 6.5 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 760 KiB/s wr, 13 op/s 2024-03-24T11:33:05.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:05 smithi179 ceph-mon[37907]: pgmap v597: 168 pgs: 168 active+clean; 6.5 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 760 KiB/s wr, 13 op/s 2024-03-24T11:33:05.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:05.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:05 smithi181 ceph-mon[33998]: pgmap v597: 168 pgs: 168 active+clean; 6.5 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 760 KiB/s wr, 13 op/s 2024-03-24T11:33:05.676 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:05.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:05.706 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:33:05.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:33:05.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:05.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:05.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:05.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:05.960 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:05.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:06.061 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:06.106 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:06.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:06.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:06.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:06.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:06.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:06.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:06.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:06.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:06 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:06 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:06 smithi181 ceph-mon[33998]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T11:33:06.482 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:06.518 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:06.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:06.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:06.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 3 15 9603 9588 199.726 186.438 0.00441549 0.00499788 2024-03-24T11:33:06.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:06.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:06 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:06 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:06 smithi179 ceph-mon[29608]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T11:33:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:06 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:06 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:06 smithi179 ceph-mon[37907]: osdmap e272: 8 total, 8 up, 8 in 2024-03-24T11:33:06.760 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:06.799 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:06.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:06.900 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:06.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:06.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:07.032 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:07.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:07.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:07.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:07.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:07.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:07.324 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:07.368 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:07.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:07.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:07.497 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:07 smithi179 ceph-mon[29608]: pgmap v599: 168 pgs: 168 active+clean; 28 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.4 MiB/s wr, 56 op/s 2024-03-24T11:33:07.497 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:07 smithi179 ceph-mon[37907]: pgmap v599: 168 pgs: 168 active+clean; 28 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.4 MiB/s wr, 56 op/s 2024-03-24T11:33:07.505 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:07.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:07.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:07.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 4 16 12416 12400 193.725 175.75 0.00304544 0.00515555 2024-03-24T11:33:07.648 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:07 smithi181 ceph-mon[33998]: pgmap v599: 168 pgs: 168 active+clean; 28 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.4 MiB/s wr, 56 op/s 2024-03-24T11:33:07.652 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:07.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:07.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:07.794 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:07.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:07.838 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:33:07.872 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:33:07.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:07.909 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:33:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:07.936 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:07.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:07.949 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:33:07.989 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:70675 9.3 9:c9cc7ea7:::benchmark_data_smithi179_130919_object70674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:46.720513+0000", 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 201.29088810499999, 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088364976999999997, 2024-03-24T11:33:08.016 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 70675 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720513+0000", 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720513+0000", 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720515+0000", 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.389e-06 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720576+0000", 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1713999999999999e-05 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720590+0000", 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3546999999999999e-05 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720598+0000", 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3100000000000001e-06 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720660+0000", 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1946000000000006e-05 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720818+0000", 2024-03-24T11:33:08.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015805100000000001 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.720980+0000", 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016166800000000001 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.723790+0000", 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028106799999999999 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.724314+0000", 2024-03-24T11:33:08.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052334300000000001 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808768+0000", 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084453818 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808831+0000", 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3031e-05 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:46.808878+0000", 2024-03-24T11:33:08.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7479999999999999e-05 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:71915 9.e 9:76fcee08:::benchmark_data_smithi179_130919_object71914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:47.811505+0000", 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 200.199896112, 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10257931300000001, 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.022 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 71915 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811505+0000", 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811505+0000", 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811506+0000", 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.569999999999999e-07 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811528+0000", 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1773999999999999e-05 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811538+0000", 2024-03-24T11:33:08.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8619999999999998e-06 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811545+0000", 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6149999999999999e-06 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811604+0000", 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8560999999999997e-05 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811763+0000", 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015900500000000001 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.811911+0000", 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000147751 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.814190+0000", 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022792789999999999 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.814304+0000", 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011387099999999999 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914006+0000", 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099702049000000001 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914053+0000", 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6888000000000002e-05 2024-03-24T11:33:08.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:47.914084+0000", 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1701e-05 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77611 9.d 9:b98574d0:::benchmark_data_smithi179_130919_object77610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.627362+0000", 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 195.38403911899999, 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101717734, 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.028 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77611 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627362+0000", 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627362+0000", 2024-03-24T11:33:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627367+0000", 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6120000000000001e-06 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627411+0000", 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4495e-05 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627421+0000", 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3419999999999997e-06 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627426+0000", 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6779999999999997e-06 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627484+0000", 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7936999999999998e-05 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627619+0000", 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000135122 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.627714+0000", 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4195000000000002e-05 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.629698+0000", 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019840050000000001 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.632388+0000", 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026900549999999998 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.728975+0000", 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.096586940999999996 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729031+0000", 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6331e-05 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729080+0000", 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9020999999999997e-05 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77618 9.d 9:bf5bec0b:::benchmark_data_smithi179_130919_object77617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.635819+0000", 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 195.375582683, 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.093501861000000006, 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.034 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77618 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635819+0000", 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635819+0000", 2024-03-24T11:33:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635823+0000", 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3080000000000001e-06 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635868+0000", 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5132000000000001e-05 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635878+0000", 2024-03-24T11:33:08.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6180000000000001e-06 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635883+0000", 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5099999999999998e-06 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.635921+0000", 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7871999999999999e-05 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.636013+0000", 2024-03-24T11:33:08.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1605000000000001e-05 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.636108+0000", 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5260999999999995e-05 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.637950+0000", 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001841996 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639110+0000", 2024-03-24T11:33:08.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00115979 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729216+0000", 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090106674999999997 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729271+0000", 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4429999999999999e-05 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.729320+0000", 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9663999999999997e-05 2024-03-24T11:33:08.039 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:77619 9.d 9:b95bc5b0:::benchmark_data_smithi179_130919_object77618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:29:52.639948+0000", 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 195.37145311399999, 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098236130000000005, 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 77619 2024-03-24T11:33:08.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639948+0000", 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639948+0000", 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.639949+0000", 2024-03-24T11:33:08.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:08.043 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:33:08.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7100000000000001e-07 2024-03-24T11:33:08.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640006+0000", 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6892999999999999e-05 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640015+0000", 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1929999999999999e-06 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640021+0000", 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6130000000000003e-06 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640051+0000", 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0815000000000002e-05 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640136+0000", 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4977999999999999e-05 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.640226+0000", 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9808999999999993e-05 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.641736+0000", 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015096440000000001 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.643416+0000", 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016803429999999999 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738077+0000", 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094660456000000004 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738135+0000", 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8733999999999997e-05 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:29:52.738184+0000", 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8980999999999997e-05 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101783 9.3 9:cd124f41:::benchmark_data_smithi179_130919_object101782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:33:08.047 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.089560+0000", 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 175.921840793, 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101679564, 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101783 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089560+0000", 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089560+0000", 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089562+0000", 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.037e-06 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089594+0000", 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2271999999999999e-05 2024-03-24T11:33:08.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089604+0000", 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0200000000000001e-05 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089610+0000", 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6590000000000004e-06 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089660+0000", 2024-03-24T11:33:08.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9917999999999999e-05 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089793+0000", 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013369400000000001 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.089967+0000", 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017335199999999999 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092374+0000", 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024076240000000001 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.095491+0000", 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031172309999999998 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191102+0000", 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.095610076000000002 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191189+0000", 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7107999999999993e-05 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191240+0000", 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1393000000000001e-05 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101786 9.3 9:c2e50cf1:::benchmark_data_smithi179_130919_object101785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:33:08.053 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.092637+0000", 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 175.91876457999999, 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098767010000000002, 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101786 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092637+0000", 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092637+0000", 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092638+0000", 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4900000000000004e-07 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092685+0000", 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7375e-05 2024-03-24T11:33:08.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092697+0000", 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2464e-05 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092706+0000", 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1340000000000004e-06 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092771+0000", 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.491e-05 2024-03-24T11:33:08.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.092913+0000", 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014212599999999999 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.093070+0000", 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015778400000000001 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.095534+0000", 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024635989999999999 2024-03-24T11:33:08.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096623+0000", 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001088583 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191279+0000", 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094656377999999999 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191365+0000", 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5792999999999997e-05 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191404+0000", 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8915000000000003e-05 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101795 9.3 9:cf9e19dc:::benchmark_data_smithi179_130919_object101794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:33:08.059 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.096719+0000", 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 175.91468244999999, 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094862295999999999, 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101795 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096719+0000", 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096719+0000", 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096729+0000", 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7540000000000001e-06 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096798+0000", 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9146999999999994e-05 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096813+0000", 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5288999999999999e-05 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096821+0000", 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4530000000000002e-06 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.096876+0000", 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5024e-05 2024-03-24T11:33:08.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.097004+0000", 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012798700000000001 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.097143+0000", 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013871200000000001 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.099660+0000", 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025171 2024-03-24T11:33:08.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.103339+0000", 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003678527 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191472+0000", 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088132919000000004 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191546+0000", 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3770999999999996e-05 2024-03-24T11:33:08.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191581+0000", 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5612999999999999e-05 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:101798 9.3 9:ca9b5007:::benchmark_data_smithi179_130919_object101797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:12.100748+0000", 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 175.910653263, 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090983586000000005, 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.065 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 101798 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100748+0000", 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100748+0000", 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100749+0000", 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2759999999999999e-06 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100828+0000", 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8549999999999998e-05 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100844+0000", 2024-03-24T11:33:08.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5676999999999999e-05 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100851+0000", 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1420000000000004e-06 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.100897+0000", 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6227999999999999e-05 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.101005+0000", 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107911 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.101104+0000", 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9347999999999998e-05 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.103427+0000", 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002322668 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.107710+0000", 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042832879999999997 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191637+0000", 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.083927392000000003 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191689+0000", 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1187000000000003e-05 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:12.191732+0000", 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2919e-05 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:107702 9.3 9:c1a39389:::benchmark_data_smithi179_130919_object107701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:16.922217+0000", 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 171.08918422100001, 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110921906, 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107702 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922217+0000", 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922217+0000", 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922218+0000", 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0210000000000001e-06 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922284+0000", 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5871999999999995e-05 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922298+0000", 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4426e-05 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922310+0000", 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.186e-05 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922402+0000", 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1268999999999996e-05 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922534+0000", 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000132189 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922683+0000", 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014913799999999999 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.927103+0000", 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044198229999999998 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.984535+0000", 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057432597000000002 2024-03-24T11:33:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.033031+0000", 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.048495908999999997 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.033091+0000", 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0149000000000003e-05 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.033139+0000", 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7652999999999999e-05 2024-03-24T11:33:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:107703 9.d 9:b4c1b45f:::benchmark_data_smithi179_130919_object107702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:16.922332+0000", 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 171.089068865, 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111412209, 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107703 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922332+0000", 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922332+0000", 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922333+0000", 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4000000000000002e-07 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922359+0000", 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6134000000000001e-05 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922365+0000", 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3319999999999997e-06 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922369+0000", 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0960000000000003e-06 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922413+0000", 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3171999999999999e-05 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922546+0000", 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013336199999999999 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922703+0000", 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015685300000000001 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.927126+0000", 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044227930000000004 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930791+0000", 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036657959999999998 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.082 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:33:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.033631+0000", 2024-03-24T11:33:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102839984 2024-03-24T11:33:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.033701+0000", 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9190000000000007e-05 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.033745+0000", 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4057000000000002e-05 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:107704 9.e 9:7f8c59d6:::benchmark_data_smithi179_130919_object107703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:16.922439+0000", 2024-03-24T11:33:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 171.088962546, 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.115749188, 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107704 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922439+0000", 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922439+0000", 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922440+0000", 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0100000000000003e-07 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922490+0000", 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9879999999999997e-05 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922502+0000", 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2019e-05 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922510+0000", 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.259e-06 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922565+0000", 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5211999999999999e-05 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922684+0000", 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011852499999999999 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.922815+0000", 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013161299999999999 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.931296+0000", 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084809840000000004 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.931309+0000", 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2571e-05 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.038112+0000", 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106803646 2024-03-24T11:33:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.038147+0000", 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4764000000000001e-05 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.038188+0000", 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0813999999999999e-05 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:107707 9.e 9:7d00fe1f:::benchmark_data_smithi179_130919_object107706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:16.925905+0000", 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 171.08549598600001, 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.116743348, 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107707 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.925905+0000", 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.925905+0000", 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.925907+0000", 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.536e-06 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.925940+0000", 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999997e-05 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.925953+0000", 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2656e-05 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.926113+0000", 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160321 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.928245+0000", 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021313819999999998 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.928402+0000", 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000157393 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.928570+0000", 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016818399999999999 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.931878+0000", 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003307562 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.931889+0000", 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0689999999999999e-05 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.042516+0000", 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11062690999999999 2024-03-24T11:33:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.042601+0000", 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5688999999999999e-05 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.042649+0000", 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7425000000000001e-05 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15357.0:107711 9.3 9:c7873d8a:::benchmark_data_smithi179_130919_object107710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:30:16.930837+0000", 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 171.08056432699999, 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11180633199999999, 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15357", 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3335304326", 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 107711 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930837+0000", 2024-03-24T11:33:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930837+0000", 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930838+0000", 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.44e-06 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930885+0000", 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7024000000000002e-05 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930898+0000", 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2734999999999999e-05 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930906+0000", 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6279999999999996e-06 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.930973+0000", 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6910999999999997e-05 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.931203+0000", 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000230005 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.931388+0000", 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000185212 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.933395+0000", 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00200746 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:16.988881+0000", 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.055486102000000002 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.042469+0000", 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053587954 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.042531+0000", 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1554000000000001e-05 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:30:17.042643+0000", 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011230699999999999 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:151296 9.9 e249/211 9:90c92071:::benchmark_data_smithi179_130919_object90512:head v 249'13728, mlcod=249'13723)", 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:42.776524+0000", 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 85.234877124999997, 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.055948405, 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.776524+0000", 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.776524+0000", 2024-03-24T11:33:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.776525+0000", 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2300000000000001e-07 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.776530+0000", 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0130000000000001e-06 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.776537+0000", 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0670000000000003e-06 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.776542+0000", 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3319999999999999e-06 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.777946+0000", 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014037349999999999 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.777962+0000", 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5936000000000001e-05 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.832379+0000", 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.054417739 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.832473+0000", 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3059999999999993e-05 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:151297 9.9 e249/211 9:90c930c5:::benchmark_data_smithi179_130919_object106998:head v 249'13729, mlcod=249'13723)", 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:42.779952+0000", 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 85.231449479999995, 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052579780999999999, 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.779952+0000", 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.779952+0000", 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.779952+0000", 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.92e-07 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.779959+0000", 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.494e-06 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.779970+0000", 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0876000000000001e-05 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.779977+0000", 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1539999999999996e-06 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.783096+0000", 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031189590000000001 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.783116+0000", 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0029000000000001e-05 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.832477+0000", 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049360714 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.832532+0000", 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4863e-05 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(client.15357.0:216582 9.1 e249/211 9:8028bd86:::benchmark_data_smithi179_130919_object82995:head v 249'13629, mlcod=249'13624)", 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:42.782143+0000", 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 85.229258302000005, 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.053528631, 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.782143+0000", 2024-03-24T11:33:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.782143+0000", 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.782144+0000", 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.06e-07 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.782148+0000", 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5979999999999999e-06 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.782157+0000", 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9379999999999998e-06 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.782163+0000", 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9290000000000003e-06 2024-03-24T11:33:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.783127+0000", 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00096359799999999997 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.783145+0000", 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8003999999999999e-05 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.835619+0000", 2024-03-24T11:33:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052474123999999997 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.835672+0000", 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2734e-05 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:151300 9.9 e249/211 9:90cc69df:::benchmark_data_smithi179_130919_object107243:head v 249'13731, mlcod=249'13723)", 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:42.784932+0000", 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 85.226469707000007, 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.052857314000000002, 2024-03-24T11:33:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.784932+0000", 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.784932+0000", 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.784932+0000", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2300000000000001e-07 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.784938+0000", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3859999999999998e-06 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.784945+0000", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8110000000000003e-06 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.784951+0000", 2024-03-24T11:33:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.784e-06 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.788892+0000", 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003941272 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.788911+0000", 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8314000000000001e-05 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.837746+0000", 2024-03-24T11:33:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.048835419999999997 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:42.837789+0000", 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2704000000000003e-05 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:152558 9.9 e250/211 9:93865b55:::benchmark_data_smithi179_130919_object130612:head v 250'14491, mlcod=250'14471)", 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:46.136199+0000", 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.875202117000001, 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.049828053999999997, 2024-03-24T11:33:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.136199+0000", 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.136199+0000", 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.136200+0000", 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5799999999999998e-07 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.136207+0000", 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5880000000000003e-06 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.136219+0000", 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2703e-05 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.136226+0000", 2024-03-24T11:33:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8419999999999999e-06 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.145197+0000", 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089711800000000005 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.145243+0000", 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5840000000000002e-05 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.185855+0000", 2024-03-24T11:33:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.040612377999999998 2024-03-24T11:33:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.186027+0000", 2024-03-24T11:33:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:08.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:08.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017176499999999999 2024-03-24T11:33:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop(osd.0.0:152560 9.9 e250/211 9:9386ae63:::benchmark_data_smithi179_130919_object130993:head v 250'14492, mlcod=250'14471)", 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:31:46.138711+0000", 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 81.872689808999993, 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.047373145999999998, 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.138711+0000", 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.138711+0000", 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.138712+0000", 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4299999999999999e-07 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.138716+0000", 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9249999999999997e-06 2024-03-24T11:33:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.138725+0000", 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8109999999999997e-06 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.138730+0000", 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3209999999999998e-06 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.148827+0000", 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010097371000000001 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.148877+0000", 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9478000000000003e-05 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.186031+0000", 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.03715454 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:31:46.186085+0000", 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3457e-05 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:33:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:33:08.130 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:33:08.130 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:33:08.130 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:33:08.130 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:33:08.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:08.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:08.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:08.358 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:08.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:08.412 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:08.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:08.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:08.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:08.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:08.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 5 16 14823 14807 185.065 150.438 0.00378117 0.00539755 2024-03-24T11:33:08.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:08.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:08.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:08.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:08 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2447499644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:08 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2447499644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:08.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:08.825 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:08.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:08 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2447499644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:08.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:08.965 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:08.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:09.067 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:09.106 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:09.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:09.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:09.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:09.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:09.266 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:33:09.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:09.397 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:09.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:09.436 INFO:teuthology.orchestra.run.smithi179.stderr:set 4.6 pg_upmap mapping to [5,2,3] 2024-03-24T11:33:09.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:09.531 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:09.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:09.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 6 15 17388 17373 180.948 160.375 0.00369851 0.00552039 2024-03-24T11:33:09.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:09.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:09 smithi181 ceph-mon[33998]: pgmap v600: 168 pgs: 168 active+clean; 195 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 24 MiB/s wr, 389 op/s 2024-03-24T11:33:09.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2139526758' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]: dispatch 2024-03-24T11:33:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]: dispatch 2024-03-24T11:33:09.668 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:09.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:09 smithi179 ceph-mon[29608]: pgmap v600: 168 pgs: 168 active+clean; 195 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 24 MiB/s wr, 389 op/s 2024-03-24T11:33:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2139526758' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]: dispatch 2024-03-24T11:33:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]: dispatch 2024-03-24T11:33:09.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:09 smithi179 ceph-mon[37907]: pgmap v600: 168 pgs: 168 active+clean; 195 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 24 MiB/s wr, 389 op/s 2024-03-24T11:33:09.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2139526758' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]: dispatch 2024-03-24T11:33:09.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]: dispatch 2024-03-24T11:33:09.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:09.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:09.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:09.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:09.944 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:09.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:10.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:10.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:10.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:10.185 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:10.222 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:10.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:10.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:10.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:10.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:10.461 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:10.502 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:10.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:10.604 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:10.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 7 16 20108 20092 179.373 169.938 0.00666244 0.00556803 2024-03-24T11:33:10.646 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:10.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:10.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:10 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]': finished 2024-03-24T11:33:10.661 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:10 smithi179 ceph-mon[29608]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T11:33:10.661 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:10 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]': finished 2024-03-24T11:33:10.661 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:10 smithi179 ceph-mon[37907]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T11:33:10.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:10.784 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:10.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:10.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:10.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:10 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 2, 3]}]': finished 2024-03-24T11:33:10.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:10 smithi181 ceph-mon[33998]: osdmap e273: 8 total, 8 up, 8 in 2024-03-24T11:33:10.909 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:10.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:10.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:11.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:11.049 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:11.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:11.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:11.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:33:11.189 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:11.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:11.292 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:11.335 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:11.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:11.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:11.437 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:11.483 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:11.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:11.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:11.614 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:11.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:11.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 8 15 22235 22220 173.575 133 0.0042872 0.00575538 2024-03-24T11:33:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:11 smithi179 ceph-mon[29608]: pgmap v602: 168 pgs: 168 active+clean; 919 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-24T11:33:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:11 smithi179 ceph-mon[29608]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T11:33:11.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:11 smithi179 ceph-mon[37907]: pgmap v602: 168 pgs: 168 active+clean; 919 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-24T11:33:11.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:11 smithi179 ceph-mon[37907]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T11:33:11.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:11.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:11.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:11.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:11.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:11.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:11 smithi181 ceph-mon[33998]: pgmap v602: 168 pgs: 168 active+clean; 919 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-24T11:33:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:11 smithi181 ceph-mon[33998]: osdmap e274: 8 total, 8 up, 8 in 2024-03-24T11:33:11.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:12.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:12.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:12.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:12.184 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:12.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:12.239 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:12.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:12.309 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:12.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:12.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:12.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:12.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:12.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:12.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:12.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:12.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 9 16 24532 24516 170.232 143.5 0.00506403 0.00586946 2024-03-24T11:33:12.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:12.726 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:12.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:12.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:12.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:12.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:12.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:12.995 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:13.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:13.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:13.120 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:13.162 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:13.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:13.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:13.297 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:13.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:13.399 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:13.441 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:13.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:13.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:13.580 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:13.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:13.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 10 16 26893 26877 167.964 147.562 0.0037913 0.0059488 2024-03-24T11:33:13.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:13 smithi179 ceph-mon[29608]: pgmap v604: 168 pgs: 168 active+clean; 919 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 738 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-24T11:33:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:13 smithi179 ceph-mon[37907]: pgmap v604: 168 pgs: 168 active+clean; 919 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 738 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-24T11:33:13.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:13.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:13.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:13.866 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:13.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:13.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:13 smithi181 ceph-mon[33998]: pgmap v604: 168 pgs: 168 active+clean; 919 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 738 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-24T11:33:13.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:14.013 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:14.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:14.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:33:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:33:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:14.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:14.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.296 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:14.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:14.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:14.433 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:14.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:14.586 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:14.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 11 16 29056 29040 164.983 135.188 0.0110367 0.00605558 2024-03-24T11:33:14.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:14.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:14.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:14.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:14.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:14.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:14.985 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:14.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:15.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:15.113 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:15.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:15.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:15.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:15.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:15.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:15.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:15.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:15.449 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:33:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:33:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:33:15.475 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:33:15.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:15.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:15.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:15.621 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 12 16 30955 30939 161.124 118.688 0.00498714 0.00619117 2024-03-24T11:33:15.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:15.650 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:15.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:15 smithi179 ceph-mon[37907]: pgmap v605: 168 pgs: 168 active+clean; 1.5 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 194 MiB/s wr, 3.10k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:15 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3555145692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:33:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:15 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:33:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:15 smithi179 ceph-mon[29608]: pgmap v605: 168 pgs: 168 active+clean; 1.5 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 194 MiB/s wr, 3.10k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:15 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3555145692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:33:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:15 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:33:15.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:15.787 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:15.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:15.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:15 smithi181 ceph-mon[33998]: pgmap v605: 168 pgs: 168 active+clean; 1.5 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 194 MiB/s wr, 3.10k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:15 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3555145692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:33:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:15 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:33:15.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:15.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:16.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:16.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:16.061 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:16.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:16.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:16.201 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:16.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:16.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:16.338 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:16.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:16.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:16.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:16.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:16.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:16.617 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:16.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:16.620 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 13 16 32811 32795 157.652 116 0.00511345 0.0063379 2024-03-24T11:33:16.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:33:16.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:33:16.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[37907]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T11:33:16.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:16.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:33:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:33:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[29608]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T11:33:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:16 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:16.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:16.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:16.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:16.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:16.899 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:16.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:16.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:16 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:33:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:16 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:33:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:16 smithi181 ceph-mon[33998]: osdmap e275: 8 total, 8 up, 8 in 2024-03-24T11:33:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:16 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:16 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:17.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:17.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:17.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:17.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:17.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:17.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:17.182 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:33:17.183 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:33:17.183 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-24T11:33:17.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:33:17.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:17.315 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:17.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:17.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:17.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:17.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:17.475 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:17.475 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:17.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:17.592 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:17.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:17.621 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 14 16 34800 34784 155.27 124.312 0.0111513 0.00643527 2024-03-24T11:33:17.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:17.729 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:17.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:17.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:17.868 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:17.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:17 smithi181 ceph-mon[33998]: pgmap v607: 168 pgs: 168 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 743 B/s rd, 109 MiB/s wr, 1.75k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:17 smithi181 ceph-mon[33998]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T11:33:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[29608]: pgmap v607: 168 pgs: 168 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 743 B/s rd, 109 MiB/s wr, 1.75k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[29608]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T11:33:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[37907]: pgmap v607: 168 pgs: 168 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 743 B/s rd, 109 MiB/s wr, 1.75k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:17 smithi179 ceph-mon[37907]: osdmap e276: 8 total, 8 up, 8 in 2024-03-24T11:33:17.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:18.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:18.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:18.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:18.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:18.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:18.244 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:18.281 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:18.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:18.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:18.418 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:18.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:18.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:18.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:18.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:18.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:18.621 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 15 16 36632 36616 152.552 114.5 0.00919609 0.00654971 2024-03-24T11:33:18.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:18.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:18.659 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:18.698 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:18.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:18.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:18.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:33:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:18.844 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:18.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:18.871 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:33:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:18.909 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:33:18.945 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:18.948 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:33:18.993 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:33:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:19.016 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:19.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:19.031 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:33:19.074 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:33:19.115 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:33:19.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:19.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:19.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:19.155 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:33:19.160 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:33:19.160 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:33:19.160 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:33:19.160 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:33:19.160 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:33:19.160 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:33:19.160 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:33:19.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:33:19.254 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:19.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:19.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:19.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:19.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:19.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:19.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:19.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:19.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:19.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 16 16 38459 38443 150.146 114.188 0.0156199 0.0066552 2024-03-24T11:33:19.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:19.716 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:19.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:19.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:19.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:33:19.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:33:19.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[29608]: pgmap v609: 168 pgs: 168 active+clean; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 650 B/s rd, 109 MiB/s wr, 1.75k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:19.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:33:19.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:33:19.871 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/676626369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:19.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:33:19.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:33:19.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[37907]: pgmap v609: 168 pgs: 168 active+clean; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 650 B/s rd, 109 MiB/s wr, 1.75k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:19.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:33:19.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:33:19.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/676626369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:19.875 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:19.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:19.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:19 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:33:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:19 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:33:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:19 smithi181 ceph-mon[33998]: pgmap v609: 168 pgs: 168 active+clean; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 650 B/s rd, 109 MiB/s wr, 1.75k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:19 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:33:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:19 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:33:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/676626369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:19.976 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:20.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:20.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:20.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:20.156 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:20.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:20.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:20.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:20.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:20.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:20.434 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:20.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:20.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:20.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:20.571 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:20.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:20.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 17 16 40383 40367 148.386 120.25 0.00434998 0.00673365 2024-03-24T11:33:20.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:20.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:20.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:20.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:33:20.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:33:20.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[29608]: 4.14 scrub starts 2024-03-24T11:33:20.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[29608]: 4.14 scrub ok 2024-03-24T11:33:20.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:33:20.756 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:33:20.757 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:33:20.757 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:33:20.757 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[37907]: 4.14 scrub starts 2024-03-24T11:33:20.757 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[37907]: 4.14 scrub ok 2024-03-24T11:33:20.757 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:33:20.757 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:20 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:33:20.812 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:20.853 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:20.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:20.869 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:20 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:33:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:20 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:33:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:20 smithi181 ceph-mon[33998]: 4.14 scrub starts 2024-03-24T11:33:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:20 smithi181 ceph-mon[33998]: 4.14 scrub ok 2024-03-24T11:33:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:20 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:33:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:20 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:33:20.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:20.960 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:20.969 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:40957 10.8 10:1681f7c7:::benchmark_data_smithi179_155357_object40956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:33:20.951992+0000", 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.016945775, 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016985257, 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:33:20.970 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 40957 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.951992+0000", 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.951992+0000", 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.951993+0000", 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.246e-06 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.952012+0000", 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9449e-05 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.952029+0000", 2024-03-24T11:33:20.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6847999999999999e-05 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.952038+0000", 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.081e-06 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.965650+0000", 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013612208000000001 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:33:20.973 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.965814+0000", 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016332599999999999 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.965979+0000", 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000165057 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968192+0000", 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0022126569999999998 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:20.974 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:40975 10.4 10:29828ed2:::benchmark_data_smithi179_155357_object40974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:33:20.968163+0000", 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00077394599999999997, 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00090821599999999999, 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "waiting for sub ops", 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 40975 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.975 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968163+0000", 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968163+0000", 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968165+0000", 2024-03-24T11:33:20.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5179999999999999e-06 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968184+0000", 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9035999999999999e-05 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968193+0000", 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0259999999999996e-06 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:33:20.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968200+0000", 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1740000000000003e-06 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968258+0000", 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7887999999999999e-05 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968425+0000", 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016728199999999999 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,4", 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968582+0000", 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015675899999999999 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.15621.0:40976 10.1 e276/267 10:8668b1ee:::benchmark_data_smithi179_155357_object40975:head v 276'2526, mlcod=276'2524)", 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:33:20.968472+0000", 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.00046537899999999998, 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00066988499999999995, 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:33:20.979 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968472+0000", 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968472+0000", 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968473+0000", 2024-03-24T11:33:20.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5099999999999998e-07 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968503+0000", 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9728999999999999e-05 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968513+0000", 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0669999999999999e-05 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:33:20.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968520+0000", 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8199999999999999e-06 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968637+0000", 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011637399999999999 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:20.968659+0000", 2024-03-24T11:33:20.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2328e-05 2024-03-24T11:33:20.983 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:20.983 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:33:20.983 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:20.983 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:20.983 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:33:20.983 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 3 2024-03-24T11:33:20.983 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:33:21.010 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:21.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:21.156 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:21.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:21.296 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:21.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:21.436 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:21.538 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:21.576 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:21.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 18 16 42111 42095 146.142 108 0.00612742 0.00683771 2024-03-24T11:33:21.677 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:21.701 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:21.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:21.841 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:21.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 1.2 scrub starts 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 1.2 scrub ok 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: pgmap v610: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:33:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:21 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:33:21.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:33:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 1.2 scrub starts 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 1.2 scrub ok 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: pgmap v610: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 1.2 scrub starts 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 1.2 scrub ok 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:33:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:33:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: pgmap v610: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:33:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:21 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:33:21.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:21.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:22.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:22.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:22.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:22.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:22.259 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:22.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:22.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:22.405 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:22.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:22.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:22.544 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:22.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:22.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 19 16 43846 43830 144.158 108.438 0.00611323 0.00693082 2024-03-24T11:33:22.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:22.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:33:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:22.685 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:22.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:22.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[29608]: 1.c scrub starts 2024-03-24T11:33:22.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[29608]: 1.c scrub ok 2024-03-24T11:33:22.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:33:22.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:33:22.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[37907]: 1.c scrub starts 2024-03-24T11:33:22.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[37907]: 1.c scrub ok 2024-03-24T11:33:22.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:33:22.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:22 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:33:22.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:22.814 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:22.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:22 smithi181 ceph-mon[33998]: 1.c scrub starts 2024-03-24T11:33:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:22 smithi181 ceph-mon[33998]: 1.c scrub ok 2024-03-24T11:33:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:22 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:33:22.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:22 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:33:22.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:22.946 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:22.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:23.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:23.087 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:23.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:23.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:23.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:23.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:23.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:23.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:23.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:23.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:23.503 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:23.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:23.605 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:23.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:33:23.622857+0000 min lat: 0.00124828 max lat: 0.147384 avg lat: 0.00704835 2024-03-24T11:33:23.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:33:23.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 20 16 45393 45377 141.784 96.6875 0.00606863 0.00704835 2024-03-24T11:33:23.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:23.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:23.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:23.797 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:23.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:23.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:23 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:33:23.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:23 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:33:23.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:23 smithi181 ceph-mon[33998]: pgmap v611: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:33:23.881 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:23 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:33:23.882 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:23 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:33:23.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:23.942 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:23.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[29608]: pgmap v611: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[37907]: pgmap v611: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:33:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:23 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:33:24.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:24.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:33:24.089 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:24.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:24.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:33:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:33:24.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:24.224 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:24.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:24.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:24.374 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:24.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:24.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:24.522 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:24.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:24.622 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 21 16 47041 47025 139.937 103 0.00858361 0.00714136 2024-03-24T11:33:24.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:24.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:24.673 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:24.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:33:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:33:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[29608]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T11:33:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:24.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[29608]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T11:33:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:33:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:33:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[37907]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T11:33:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:24.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:24 smithi179 ceph-mon[37907]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T11:33:24.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:24.810 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:24.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:24 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:33:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:24 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:33:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:24 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:24 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:24 smithi181 ceph-mon[33998]: osdmap e277: 8 total, 8 up, 8 in 2024-03-24T11:33:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:24 smithi181 ceph-mon[33998]: osdmap e278: 8 total, 8 up, 8 in 2024-03-24T11:33:24.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:24.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:24.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:25.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:25.089 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:25.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:25.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:25.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:25.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:25.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:25.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:25.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:25.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:33:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:33:25.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:25.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:25.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:25.623 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 22 16 48756 48740 138.448 107.188 0.00589982 0.00721796 2024-03-24T11:33:25.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:25.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:25.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[29608]: pgmap v614: 168 pgs: 168 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:33:25.729 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:33:25.730 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:33:25.730 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[37907]: pgmap v614: 168 pgs: 168 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-24T11:33:25.730 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:33:25.730 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:33:25.730 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:33:25.730 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:25 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:33:25.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:25.810 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:25.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:25.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:25 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:33:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:25 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:33:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:25 smithi181 ceph-mon[33998]: pgmap v614: 168 pgs: 168 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-24T11:33:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:25 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:33:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:25 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:33:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:25 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:33:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:25 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:33:25.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:25.948 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:25.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:26.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:26.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:26.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:26.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:26.217 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:26.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:26.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:26.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:26.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:26.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:26.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:26.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:26.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:26.622 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:26.623 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:26.623 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 23 16 50175 50159 136.28 88.6875 0.0141959 0.00733242 2024-03-24T11:33:26.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:26.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:26.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:26.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:26.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:33:26.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:33:26.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[29608]: 3.b scrub starts 2024-03-24T11:33:26.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[29608]: 3.b scrub ok 2024-03-24T11:33:26.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:33:26.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:33:26.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[37907]: 3.b scrub starts 2024-03-24T11:33:26.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:26 smithi179 ceph-mon[37907]: 3.b scrub ok 2024-03-24T11:33:26.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:26.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:26 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:33:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:26 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:33:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:26 smithi181 ceph-mon[33998]: 3.b scrub starts 2024-03-24T11:33:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:26 smithi181 ceph-mon[33998]: 3.b scrub ok 2024-03-24T11:33:26.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:26.961 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:26.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:27.061 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:27.096 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:33:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:27.106 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:27.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:27.112 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:33:27.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:27.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:27.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:27.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:27.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:27.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:27.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:27.521 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:27.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:27.575 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:33:27.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:27.624 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 24 16 51748 51732 134.698 98.3125 0.0105823 0.00741901 2024-03-24T11:33:27.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:27.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:27.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:27.795 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:27.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:27.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: pgmap v615: 168 pgs: 168 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2816353893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: 6.1c scrub starts 2024-03-24T11:33:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:27 smithi181 ceph-mon[33998]: 6.1c scrub ok 2024-03-24T11:33:27.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:33:27.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:33:27.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: pgmap v615: 168 pgs: 168 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-24T11:33:27.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:33:27.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:33:27.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2816353893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:33:27.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: 6.1c scrub starts 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[29608]: 6.1c scrub ok 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: pgmap v615: 168 pgs: 168 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2816353893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: 6.1c scrub starts 2024-03-24T11:33:27.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:27 smithi179 ceph-mon[37907]: 6.1c scrub ok 2024-03-24T11:33:27.933 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:27.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:28.035 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:28.079 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:28.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:28.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:28.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:28.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:28.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:28.364 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2024-03-24T11:33:26.315945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2968656222,"num_objects":45487,"num_object_clones":0,"num_object_copies":136461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45487,"num_whiteouts":0,"num_read":1325,"num_read_kb":1077,"num_write":46011,"num_write_kb":2898541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48801,"ondisk_log_size":48801,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12236936,"kb_used_data":8630592,"kb_used_omap":12,"kb_used_meta":3606259,"kb_avail":737724280,"statfs":{"total":767960285184,"available":755429662720,"internally_reserved":0,"allocated":8837726208,"data_stored":8834886283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3692809777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,5,2,6,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1100218368,"num_objects":16788,"num_object_clones":0,"num_object_copies":50364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16788,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16788,"num_write_kb":1074432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004092"},"pg_stats":[{"pgid":"10.2","version":"277'2859","reported_seq":2889,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.000383+0000","last_change":"2024-03-24T11:32:52.328191+0000","last_active":"2024-03-24T11:33:24.000383+0000","last_peered":"2024-03-24T11:33:24.000383+0000","last_clean":"2024-03-24T11:33:24.000383+0000","last_became_active":"2024-03-24T11:32:52.327879+0000","last_became_peered":"2024-03-24T11:32:52.327879+0000","last_unstale":"2024-03-24T11:33:24.000383+0000","last_undegraded":"2024-03-24T11:33:24.000383+0000","last_fullsized":"2024-03-24T11:33:24.000383+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2859,"log_dups_size":0,"ondisk_log_size":2859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:12.127118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187367424,"num_objects":2859,"num_object_clones":0,"num_object_copies":8577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":182976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"213'20","reported_seq":511,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.484417+0000","last_change":"2024-03-24T11:32:36.799202+0000","last_active":"2024-03-24T11:33:16.484417+0000","last_peered":"2024-03-24T11:33:16.484417+0000","last_clean":"2024-03-24T11:33:16.484417+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:33:16.484417+0000","last_undegraded":"2024-03-24T11:33:16.484417+0000","last_fullsized":"2024-03-24T11:33:16.484417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:36.799084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.799084+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:06.179015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":473,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237844+0000","last_change":"2024-03-24T11:29:27.048143+0000","last_active":"2024-03-24T11:33:24.237844+0000","last_peered":"2024-03-24T11:33:24.237844+0000","last_clean":"2024-03-24T11:33:24.237844+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:33:24.237844+0000","last_undegraded":"2024-03-24T11:33:24.237844+0000","last_fullsized":"2024-03-24T11:33:24.237844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:27.048023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:29:27.048023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":685,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530862+0000","last_change":"2024-03-24T11:32:37.091705+0000","last_active":"2024-03-24T11:33:23.530862+0000","last_peered":"2024-03-24T11:33:23.530862+0000","last_clean":"2024-03-24T11:33:23.530862+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:33:23.530862+0000","last_undegraded":"2024-03-24T11:33:23.530862+0000","last_fullsized":"2024-03-24T11:33:23.530862+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:37.091630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.091630+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.785943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":473,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.485486+0000","last_change":"2024-03-24T11:31:15.842164+0000","last_active":"2024-03-24T11:33:16.485486+0000","last_peered":"2024-03-24T11:33:16.485486+0000","last_clean":"2024-03-24T11:33:16.485486+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:33:16.485486+0000","last_undegraded":"2024-03-24T11:33:16.485486+0000","last_fullsized":"2024-03-24T11:33:16.485486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.842054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:16.026113+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.842054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.639965+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":535,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236731+0000","last_change":"2024-03-24T11:31:10.289542+0000","last_active":"2024-03-24T11:33:24.236731+0000","last_peered":"2024-03-24T11:33:24.236731+0000","last_clean":"2024-03-24T11:33:24.236731+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:33:24.236731+0000","last_undegraded":"2024-03-24T11:33:24.236731+0000","last_fullsized":"2024-03-24T11:33:24.236731+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:10.289309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.289309+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"277'2870","reported_seq":2900,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.655706+0000","last_change":"2024-03-24T11:32:52.671692+0000","last_active":"2024-03-24T11:33:23.655706+0000","last_peered":"2024-03-24T11:33:23.655706+0000","last_clean":"2024-03-24T11:33:23.655706+0000","last_became_active":"2024-03-24T11:32:52.671360+0000","last_became_peered":"2024-03-24T11:32:52.671360+0000","last_unstale":"2024-03-24T11:33:23.655706+0000","last_undegraded":"2024-03-24T11:33:23.655706+0000","last_fullsized":"2024-03-24T11:33:23.655706+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2870,"log_dups_size":0,"ondisk_log_size":2870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:04.227723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":183680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"213'39","reported_seq":570,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.484726+0000","last_change":"2024-03-24T11:32:42.683845+0000","last_active":"2024-03-24T11:33:16.484726+0000","last_peered":"2024-03-24T11:33:16.484726+0000","last_clean":"2024-03-24T11:33:16.484726+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:33:16.484726+0000","last_undegraded":"2024-03-24T11:33:16.484726+0000","last_fullsized":"2024-03-24T11:33:16.484726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:32:42.683738+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.683738+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.142199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":516,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.481743+0000","last_change":"2024-03-24T11:32:23.906538+0000","last_active":"2024-03-24T11:33:16.481743+0000","last_peered":"2024-03-24T11:33:16.481743+0000","last_clean":"2024-03-24T11:33:16.481743+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:33:16.481743+0000","last_undegraded":"2024-03-24T11:33:16.481743+0000","last_fullsized":"2024-03-24T11:33:16.481743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:32:23.906455+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.906455+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.906085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":495,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525140+0000","last_change":"2024-03-24T11:32:25.619271+0000","last_active":"2024-03-24T11:33:23.525140+0000","last_peered":"2024-03-24T11:33:23.525140+0000","last_clean":"2024-03-24T11:33:23.525140+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:33:23.525140+0000","last_undegraded":"2024-03-24T11:33:23.525140+0000","last_fullsized":"2024-03-24T11:33:23.525140+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:25.619184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.619184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.019241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":491,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.531361+0000","last_change":"2024-03-24T11:31:35.151699+0000","last_active":"2024-03-24T11:33:23.531361+0000","last_peered":"2024-03-24T11:33:23.531361+0000","last_clean":"2024-03-24T11:33:23.531361+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:33:23.531361+0000","last_undegraded":"2024-03-24T11:33:23.531361+0000","last_fullsized":"2024-03-24T11:33:23.531361+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:35.151566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:35.151566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":630,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.527593+0000","last_change":"2024-03-24T11:31:31.621753+0000","last_active":"2024-03-24T11:33:23.527593+0000","last_peered":"2024-03-24T11:33:23.527593+0000","last_clean":"2024-03-24T11:33:23.527593+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:33:23.527593+0000","last_undegraded":"2024-03-24T11:33:23.527593+0000","last_fullsized":"2024-03-24T11:33:23.527593+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.621646+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"276'2755","reported_seq":2783,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.315210+0000","last_change":"2024-03-24T11:32:52.324098+0000","last_active":"2024-03-24T11:33:23.315210+0000","last_peered":"2024-03-24T11:33:23.315210+0000","last_clean":"2024-03-24T11:33:23.315210+0000","last_became_active":"2024-03-24T11:32:52.323807+0000","last_became_peered":"2024-03-24T11:32:52.323807+0000","last_unstale":"2024-03-24T11:33:23.315210+0000","last_undegraded":"2024-03-24T11:33:23.315210+0000","last_fullsized":"2024-03-24T11:33:23.315210+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2755,"log_dups_size":0,"ondisk_log_size":2755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:19.054452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180551680,"num_objects":2755,"num_object_clones":0,"num_object_copies":8265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2755,"num_write_kb":176320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"213'19","reported_seq":520,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.485174+0000","last_change":"2024-03-24T11:32:37.784109+0000","last_active":"2024-03-24T11:33:16.485174+0000","last_peered":"2024-03-24T11:33:16.485174+0000","last_clean":"2024-03-24T11:33:16.485174+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:33:16.485174+0000","last_undegraded":"2024-03-24T11:33:16.485174+0000","last_fullsized":"2024-03-24T11:33:16.485174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:37.784011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.784011+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:39.413955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":485,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480719+0000","last_change":"2024-03-24T11:31:29.730215+0000","last_active":"2024-03-24T11:33:16.480719+0000","last_peered":"2024-03-24T11:33:16.480719+0000","last_clean":"2024-03-24T11:33:16.480719+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:33:16.480719+0000","last_undegraded":"2024-03-24T11:33:16.480719+0000","last_fullsized":"2024-03-24T11:33:16.480719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.730078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.730078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":487,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.531971+0000","last_change":"2024-03-24T11:33:18.355226+0000","last_active":"2024-03-24T11:33:23.531971+0000","last_peered":"2024-03-24T11:33:23.531971+0000","last_clean":"2024-03-24T11:33:23.531971+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:33:23.531971+0000","last_undegraded":"2024-03-24T11:33:23.531971+0000","last_fullsized":"2024-03-24T11:33:23.531971+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:18.355107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:33:18.355107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:31.561790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":461,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238485+0000","last_change":"2024-03-24T11:29:39.057894+0000","last_active":"2024-03-24T11:33:24.238485+0000","last_peered":"2024-03-24T11:33:24.238485+0000","last_clean":"2024-03-24T11:33:24.238485+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:33:24.238485+0000","last_undegraded":"2024-03-24T11:33:24.238485+0000","last_fullsized":"2024-03-24T11:33:24.238485+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:39.057644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:39.057644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":649,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.524328+0000","last_change":"2024-03-24T11:32:24.651691+0000","last_active":"2024-03-24T11:33:23.524328+0000","last_peered":"2024-03-24T11:33:23.524328+0000","last_clean":"2024-03-24T11:33:23.524328+0000","last_became_active":"2024-03-24T11:28:43.190009+0000","last_became_peered":"2024-03-24T11:28:43.190009+0000","last_unstale":"2024-03-24T11:33:23.524328+0000","last_undegraded":"2024-03-24T11:33:23.524328+0000","last_fullsized":"2024-03-24T11:33:23.524328+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:24.651587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.651587+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.416321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"276'2783","reported_seq":2810,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.266217+0000","last_change":"2024-03-24T11:32:52.671340+0000","last_active":"2024-03-24T11:33:23.266217+0000","last_peered":"2024-03-24T11:33:23.266217+0000","last_clean":"2024-03-24T11:33:23.266217+0000","last_became_active":"2024-03-24T11:32:52.671000+0000","last_became_peered":"2024-03-24T11:32:52.671000+0000","last_unstale":"2024-03-24T11:33:23.266217+0000","last_undegraded":"2024-03-24T11:33:23.266217+0000","last_fullsized":"2024-03-24T11:33:23.266217+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2783,"log_dups_size":0,"ondisk_log_size":2783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:16.478016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182386688,"num_objects":2783,"num_object_clones":0,"num_object_copies":8349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":178112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"213'34","reported_seq":549,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525714+0000","last_change":"2024-03-24T11:32:30.766638+0000","last_active":"2024-03-24T11:33:23.525714+0000","last_peered":"2024-03-24T11:33:23.525714+0000","last_clean":"2024-03-24T11:33:23.525714+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:33:23.525714+0000","last_undegraded":"2024-03-24T11:33:23.525714+0000","last_fullsized":"2024-03-24T11:33:23.525714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:32:30.766549+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.766549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.554399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":481,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236622+0000","last_change":"2024-03-24T11:31:22.167292+0000","last_active":"2024-03-24T11:33:24.236622+0000","last_peered":"2024-03-24T11:33:24.236622+0000","last_clean":"2024-03-24T11:33:24.236622+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:33:24.236622+0000","last_undegraded":"2024-03-24T11:33:24.236622+0000","last_fullsized":"2024-03-24T11:33:24.236622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.167171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.167171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":659,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.485438+0000","last_change":"2024-03-24T11:32:18.783487+0000","last_active":"2024-03-24T11:33:16.485438+0000","last_peered":"2024-03-24T11:33:16.485438+0000","last_clean":"2024-03-24T11:33:16.485438+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:33:16.485438+0000","last_undegraded":"2024-03-24T11:33:16.485438+0000","last_fullsized":"2024-03-24T11:33:16.485438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:18.783339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.783339+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:29.190654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":475,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528266+0000","last_change":"2024-03-24T11:31:28.690070+0000","last_active":"2024-03-24T11:33:23.528266+0000","last_peered":"2024-03-24T11:33:23.528266+0000","last_clean":"2024-03-24T11:33:23.528266+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:33:23.528266+0000","last_undegraded":"2024-03-24T11:33:23.528266+0000","last_fullsized":"2024-03-24T11:33:23.528266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.689946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.689946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":629,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.527868+0000","last_change":"2024-03-24T11:33:21.341057+0000","last_active":"2024-03-24T11:33:23.527868+0000","last_peered":"2024-03-24T11:33:23.527868+0000","last_clean":"2024-03-24T11:33:23.527868+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:33:23.527868+0000","last_undegraded":"2024-03-24T11:33:23.527868+0000","last_fullsized":"2024-03-24T11:33:23.527868+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:21.340939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:21.340939+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:25.744920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"276'2844","reported_seq":2872,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.260669+0000","last_change":"2024-03-24T11:32:52.325471+0000","last_active":"2024-03-24T11:33:23.260669+0000","last_peered":"2024-03-24T11:33:23.260669+0000","last_clean":"2024-03-24T11:33:23.260669+0000","last_became_active":"2024-03-24T11:32:52.325161+0000","last_became_peered":"2024-03-24T11:32:52.325161+0000","last_unstale":"2024-03-24T11:33:23.260669+0000","last_undegraded":"2024-03-24T11:33:23.260669+0000","last_fullsized":"2024-03-24T11:33:23.260669+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2844,"log_dups_size":0,"ondisk_log_size":2844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:57.289098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186384384,"num_objects":2844,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2844,"num_write_kb":182016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":485,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480638+0000","last_change":"2024-03-24T11:32:39.715836+0000","last_active":"2024-03-24T11:33:16.480638+0000","last_peered":"2024-03-24T11:33:16.480638+0000","last_clean":"2024-03-24T11:33:16.480638+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:33:16.480638+0000","last_undegraded":"2024-03-24T11:33:16.480638+0000","last_fullsized":"2024-03-24T11:33:16.480638+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.715759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.083140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"213'30","reported_seq":520,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236954+0000","last_change":"2024-03-24T11:31:31.099444+0000","last_active":"2024-03-24T11:33:24.236954+0000","last_peered":"2024-03-24T11:33:24.236954+0000","last_clean":"2024-03-24T11:33:24.236954+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:33:24.236954+0000","last_undegraded":"2024-03-24T11:33:24.236954+0000","last_fullsized":"2024-03-24T11:33:24.236954+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:31:31.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.099209+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":473,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237612+0000","last_change":"2024-03-24T11:29:28.019634+0000","last_active":"2024-03-24T11:33:24.237612+0000","last_peered":"2024-03-24T11:33:24.237612+0000","last_clean":"2024-03-24T11:33:24.237612+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:33:24.237612+0000","last_undegraded":"2024-03-24T11:33:24.237612+0000","last_fullsized":"2024-03-24T11:33:24.237612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:28.018667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:23.433649+0000","last_clean_scrub_stamp":"2024-03-24T11:29:28.018667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":465,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479491+0000","last_change":"2024-03-24T11:30:17.941383+0000","last_active":"2024-03-24T11:33:16.479491+0000","last_peered":"2024-03-24T11:33:16.479491+0000","last_clean":"2024-03-24T11:33:16.479491+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:33:16.479491+0000","last_undegraded":"2024-03-24T11:33:16.479491+0000","last_fullsized":"2024-03-24T11:33:16.479491+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.941259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.639843+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":632,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.484307+0000","last_change":"2024-03-24T11:31:25.692917+0000","last_active":"2024-03-24T11:33:16.484307+0000","last_peered":"2024-03-24T11:33:16.484307+0000","last_clean":"2024-03-24T11:33:16.484307+0000","last_became_active":"2024-03-24T11:28:38.173086+0000","last_became_peered":"2024-03-24T11:28:38.173086+0000","last_unstale":"2024-03-24T11:33:16.484307+0000","last_undegraded":"2024-03-24T11:33:16.484307+0000","last_fullsized":"2024-03-24T11:33:16.484307+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.692790+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:28:38.972499+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.692790+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"276'2785","reported_seq":2813,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.270792+0000","last_change":"2024-03-24T11:32:52.673849+0000","last_active":"2024-03-24T11:33:23.270792+0000","last_peered":"2024-03-24T11:33:23.270792+0000","last_clean":"2024-03-24T11:33:23.270792+0000","last_became_active":"2024-03-24T11:32:52.673505+0000","last_became_peered":"2024-03-24T11:32:52.673505+0000","last_unstale":"2024-03-24T11:33:23.270792+0000","last_undegraded":"2024-03-24T11:33:23.270792+0000","last_fullsized":"2024-03-24T11:33:23.270792+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2785,"log_dups_size":0,"ondisk_log_size":2785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.935112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517760,"num_objects":2785,"num_object_clones":0,"num_object_copies":8355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2785,"num_write_kb":178240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":649,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237061+0000","last_change":"2024-03-24T11:31:08.290147+0000","last_active":"2024-03-24T11:33:24.237061+0000","last_peered":"2024-03-24T11:33:24.237061+0000","last_clean":"2024-03-24T11:33:24.237061+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:33:24.237061+0000","last_undegraded":"2024-03-24T11:33:24.237061+0000","last_fullsized":"2024-03-24T11:33:24.237061+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:08.289960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.289960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"213'28","reported_seq":523,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480815+0000","last_change":"2024-03-24T11:32:34.830458+0000","last_active":"2024-03-24T11:33:16.480815+0000","last_peered":"2024-03-24T11:33:16.480815+0000","last_clean":"2024-03-24T11:33:16.480815+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:33:16.480815+0000","last_undegraded":"2024-03-24T11:33:16.480815+0000","last_fullsized":"2024-03-24T11:33:16.480815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:32:34.830322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.830322+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.916433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":498,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.232053+0000","last_change":"2024-03-24T11:32:40.035107+0000","last_active":"2024-03-24T11:33:24.232053+0000","last_peered":"2024-03-24T11:33:24.232053+0000","last_clean":"2024-03-24T11:33:24.232053+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:33:24.232053+0000","last_undegraded":"2024-03-24T11:33:24.232053+0000","last_fullsized":"2024-03-24T11:33:24.232053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:32:40.035022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.035022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:47.238292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":105,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482379+0000","last_change":"2024-03-24T11:32:33.957761+0000","last_active":"2024-03-24T11:33:16.482379+0000","last_peered":"2024-03-24T11:33:16.482379+0000","last_clean":"2024-03-24T11:33:16.482379+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:33:16.482379+0000","last_undegraded":"2024-03-24T11:33:16.482379+0000","last_fullsized":"2024-03-24T11:33:16.482379+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:33.957632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.957632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.301310+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":628,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238351+0000","last_change":"2024-03-24T11:31:18.144752+0000","last_active":"2024-03-24T11:33:24.238351+0000","last_peered":"2024-03-24T11:33:24.238351+0000","last_clean":"2024-03-24T11:33:24.238351+0000","last_became_active":"2024-03-24T11:28:38.174888+0000","last_became_peered":"2024-03-24T11:28:38.174888+0000","last_unstale":"2024-03-24T11:33:24.238351+0000","last_undegraded":"2024-03-24T11:33:24.238351+0000","last_fullsized":"2024-03-24T11:33:24.238351+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.144322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.144322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"276'2730","reported_seq":2758,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.256770+0000","last_change":"2024-03-24T11:32:52.670936+0000","last_active":"2024-03-24T11:33:23.256770+0000","last_peered":"2024-03-24T11:33:23.256770+0000","last_clean":"2024-03-24T11:33:23.256770+0000","last_became_active":"2024-03-24T11:32:52.670591+0000","last_became_peered":"2024-03-24T11:32:52.670591+0000","last_unstale":"2024-03-24T11:33:23.256770+0000","last_undegraded":"2024-03-24T11:33:23.256770+0000","last_fullsized":"2024-03-24T11:33:23.256770+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:07.793729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":174720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":673,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.235656+0000","last_change":"2024-03-24T11:32:42.033121+0000","last_active":"2024-03-24T11:33:24.235656+0000","last_peered":"2024-03-24T11:33:24.235656+0000","last_clean":"2024-03-24T11:33:24.235656+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:33:24.235656+0000","last_undegraded":"2024-03-24T11:33:24.235656+0000","last_fullsized":"2024-03-24T11:33:24.235656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.033019+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.550557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"213'20","reported_seq":529,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525177+0000","last_change":"2024-03-24T11:32:21.588381+0000","last_active":"2024-03-24T11:33:23.525177+0000","last_peered":"2024-03-24T11:33:23.525177+0000","last_clean":"2024-03-24T11:33:23.525177+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:33:23.525177+0000","last_undegraded":"2024-03-24T11:33:23.525177+0000","last_fullsized":"2024-03-24T11:33:23.525177+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:21.588239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.588239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.355301+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":498,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528492+0000","last_change":"2024-03-24T11:31:06.261980+0000","last_active":"2024-03-24T11:33:23.528492+0000","last_peered":"2024-03-24T11:33:23.528492+0000","last_clean":"2024-03-24T11:33:23.528492+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:33:23.528492+0000","last_undegraded":"2024-03-24T11:33:23.528492+0000","last_fullsized":"2024-03-24T11:33:23.528492+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:06.261883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.261883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":479,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.231953+0000","last_change":"2024-03-24T11:32:22.994060+0000","last_active":"2024-03-24T11:33:24.231953+0000","last_peered":"2024-03-24T11:33:24.231953+0000","last_clean":"2024-03-24T11:33:24.231953+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:33:24.231953+0000","last_undegraded":"2024-03-24T11:33:24.231953+0000","last_fullsized":"2024-03-24T11:33:24.231953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:32:22.993851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:22.993851+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.104574+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":646,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482242+0000","last_change":"2024-03-24T11:31:42.062272+0000","last_active":"2024-03-24T11:33:16.482242+0000","last_peered":"2024-03-24T11:33:16.482242+0000","last_clean":"2024-03-24T11:33:16.482242+0000","last_became_active":"2024-03-24T11:28:38.178260+0000","last_became_peered":"2024-03-24T11:28:38.178260+0000","last_unstale":"2024-03-24T11:33:16.482242+0000","last_undegraded":"2024-03-24T11:33:16.482242+0000","last_fullsized":"2024-03-24T11:33:16.482242+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.062151+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"276'2814","reported_seq":2842,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.293919+0000","last_change":"2024-03-24T11:32:52.327929+0000","last_active":"2024-03-24T11:33:23.293919+0000","last_peered":"2024-03-24T11:33:23.293919+0000","last_clean":"2024-03-24T11:33:23.293919+0000","last_became_active":"2024-03-24T11:32:52.327652+0000","last_became_peered":"2024-03-24T11:32:52.327652+0000","last_unstale":"2024-03-24T11:33:23.293919+0000","last_undegraded":"2024-03-24T11:33:23.293919+0000","last_fullsized":"2024-03-24T11:33:23.293919+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2814,"log_dups_size":0,"ondisk_log_size":2814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:17.040480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":180096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":501,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482101+0000","last_change":"2024-03-24T11:31:33.075345+0000","last_active":"2024-03-24T11:33:16.482101+0000","last_peered":"2024-03-24T11:33:16.482101+0000","last_clean":"2024-03-24T11:33:16.482101+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:33:16.482101+0000","last_undegraded":"2024-03-24T11:33:16.482101+0000","last_fullsized":"2024-03-24T11:33:16.482101+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:33.075228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.075228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"213'29","reported_seq":536,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.235319+0000","last_change":"2024-03-24T11:32:26.907881+0000","last_active":"2024-03-24T11:33:24.235319+0000","last_peered":"2024-03-24T11:33:24.235319+0000","last_clean":"2024-03-24T11:33:24.235319+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:33:24.235319+0000","last_undegraded":"2024-03-24T11:33:24.235319+0000","last_fullsized":"2024-03-24T11:33:24.235319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:32:26.907769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.907769+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:32.644339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":511,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.524867+0000","last_change":"2024-03-24T11:33:20.238699+0000","last_active":"2024-03-24T11:33:23.524867+0000","last_peered":"2024-03-24T11:33:23.524867+0000","last_clean":"2024-03-24T11:33:23.524867+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:33:23.524867+0000","last_undegraded":"2024-03-24T11:33:23.524867+0000","last_fullsized":"2024-03-24T11:33:23.524867+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:20.238585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.238585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:35.247654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":491,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.523584+0000","last_change":"2024-03-24T11:32:45.851666+0000","last_active":"2024-03-24T11:33:23.523584+0000","last_peered":"2024-03-24T11:33:23.523584+0000","last_clean":"2024-03-24T11:33:23.523584+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:33:23.523584+0000","last_undegraded":"2024-03-24T11:33:23.523584+0000","last_fullsized":"2024-03-24T11:33:23.523584+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.851566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.851566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:02.916651+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":646,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.481926+0000","last_change":"2024-03-24T11:31:18.894161+0000","last_active":"2024-03-24T11:33:16.481926+0000","last_peered":"2024-03-24T11:33:16.481926+0000","last_clean":"2024-03-24T11:33:16.481926+0000","last_became_active":"2024-03-24T11:28:38.174742+0000","last_became_peered":"2024-03-24T11:28:38.174742+0000","last_unstale":"2024-03-24T11:33:16.481926+0000","last_undegraded":"2024-03-24T11:33:16.481926+0000","last_fullsized":"2024-03-24T11:33:16.481926+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.894050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.894050+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"278'2967","reported_seq":2999,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.340220+0000","last_change":"2024-03-24T11:32:52.671042+0000","last_active":"2024-03-24T11:33:24.340220+0000","last_peered":"2024-03-24T11:33:24.340220+0000","last_clean":"2024-03-24T11:33:24.340220+0000","last_became_active":"2024-03-24T11:32:52.670737+0000","last_became_peered":"2024-03-24T11:32:52.670737+0000","last_unstale":"2024-03-24T11:33:24.340220+0000","last_undegraded":"2024-03-24T11:33:24.340220+0000","last_fullsized":"2024-03-24T11:33:24.340220+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2967,"log_dups_size":0,"ondisk_log_size":2967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:01.828858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194445312,"num_objects":2967,"num_object_clones":0,"num_object_copies":8901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2967,"num_write_kb":189888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":479,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.532055+0000","last_change":"2024-03-24T11:30:27.391730+0000","last_active":"2024-03-24T11:33:23.532055+0000","last_peered":"2024-03-24T11:33:23.532055+0000","last_clean":"2024-03-24T11:33:23.532055+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:33:23.532055+0000","last_undegraded":"2024-03-24T11:33:23.532055+0000","last_fullsized":"2024-03-24T11:33:23.532055+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:30:27.391607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"213'24","reported_seq":37,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.529341+0000","last_change":"2024-03-24T11:33:10.483025+0000","last_active":"2024-03-24T11:33:23.529341+0000","last_peered":"2024-03-24T11:33:23.529341+0000","last_clean":"2024-03-24T11:33:23.529341+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:33:23.529341+0000","last_undegraded":"2024-03-24T11:33:23.529341+0000","last_fullsized":"2024-03-24T11:33:23.529341+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:21.972644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.972644+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:32.623812+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":319,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.523773+0000","last_change":"2024-03-24T11:32:39.822026+0000","last_active":"2024-03-24T11:33:23.523773+0000","last_peered":"2024-03-24T11:33:23.523773+0000","last_clean":"2024-03-24T11:33:23.523773+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:33:23.523773+0000","last_undegraded":"2024-03-24T11:33:23.523773+0000","last_fullsized":"2024-03-24T11:33:23.523773+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:32:39.821933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.821933+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.501549+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":175,"num_read_kb":170,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":500,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.229418+0000","last_change":"2024-03-24T11:33:19.182254+0000","last_active":"2024-03-24T11:33:24.229418+0000","last_peered":"2024-03-24T11:33:24.229418+0000","last_clean":"2024-03-24T11:33:24.229418+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:33:24.229418+0000","last_undegraded":"2024-03-24T11:33:24.229418+0000","last_fullsized":"2024-03-24T11:33:24.229418+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.182060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.182060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:28.529107+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":491,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.524633+0000","last_change":"2024-03-24T11:31:42.720752+0000","last_active":"2024-03-24T11:33:23.524633+0000","last_peered":"2024-03-24T11:33:23.524633+0000","last_clean":"2024-03-24T11:33:23.524633+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:33:23.524633+0000","last_undegraded":"2024-03-24T11:33:23.524633+0000","last_fullsized":"2024-03-24T11:33:23.524633+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:42.720504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:31:42.720504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":642,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.524431+0000","last_change":"2024-03-24T11:32:26.629982+0000","last_active":"2024-03-24T11:33:23.524431+0000","last_peered":"2024-03-24T11:33:23.524431+0000","last_clean":"2024-03-24T11:33:23.524431+0000","last_became_active":"2024-03-24T11:28:38.173843+0000","last_became_peered":"2024-03-24T11:28:38.173843+0000","last_unstale":"2024-03-24T11:33:23.524431+0000","last_undegraded":"2024-03-24T11:33:23.524431+0000","last_fullsized":"2024-03-24T11:33:23.524431+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:26.629894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.629894+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.767066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"276'2829","reported_seq":2857,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.293507+0000","last_change":"2024-03-24T11:32:52.328051+0000","last_active":"2024-03-24T11:33:23.293507+0000","last_peered":"2024-03-24T11:33:23.293507+0000","last_clean":"2024-03-24T11:33:23.293507+0000","last_became_active":"2024-03-24T11:32:52.327768+0000","last_became_peered":"2024-03-24T11:32:52.327768+0000","last_unstale":"2024-03-24T11:33:23.293507+0000","last_undegraded":"2024-03-24T11:33:23.293507+0000","last_fullsized":"2024-03-24T11:33:23.293507+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:18.657911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185401344,"num_objects":2829,"num_object_clones":0,"num_object_copies":8487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":181056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":107,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530630+0000","last_change":"2024-03-24T11:32:41.207872+0000","last_active":"2024-03-24T11:33:23.530630+0000","last_peered":"2024-03-24T11:33:23.530630+0000","last_clean":"2024-03-24T11:33:23.530630+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:33:23.530630+0000","last_undegraded":"2024-03-24T11:33:23.530630+0000","last_fullsized":"2024-03-24T11:33:23.530630+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:41.207792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.207792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.669676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"213'36","reported_seq":727,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482419+0000","last_change":"2024-03-24T11:32:25.853007+0000","last_active":"2024-03-24T11:33:16.482419+0000","last_peered":"2024-03-24T11:33:16.482419+0000","last_clean":"2024-03-24T11:33:16.482419+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:33:16.482419+0000","last_undegraded":"2024-03-24T11:33:16.482419+0000","last_fullsized":"2024-03-24T11:33:16.482419+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'36","last_scrub_stamp":"2024-03-24T11:32:25.852913+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.852913+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.114595+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":487,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236561+0000","last_change":"2024-03-24T11:32:28.089047+0000","last_active":"2024-03-24T11:33:24.236561+0000","last_peered":"2024-03-24T11:33:24.236561+0000","last_clean":"2024-03-24T11:33:24.236561+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:33:24.236561+0000","last_undegraded":"2024-03-24T11:33:24.236561+0000","last_fullsized":"2024-03-24T11:33:24.236561+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.088969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.088969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.266746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":405,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.232057+0000","last_change":"2024-03-24T11:33:18.225225+0000","last_active":"2024-03-24T11:33:24.232057+0000","last_peered":"2024-03-24T11:33:24.232057+0000","last_clean":"2024-03-24T11:33:24.232057+0000","last_became_active":"2024-03-24T11:30:35.355009+0000","last_became_peered":"2024-03-24T11:30:35.355009+0000","last_unstale":"2024-03-24T11:33:24.232057+0000","last_undegraded":"2024-03-24T11:33:24.232057+0000","last_fullsized":"2024-03-24T11:33:24.232057+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:18.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:18.225111+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:31.427035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":491,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.529914+0000","last_change":"2024-03-24T11:31:26.676287+0000","last_active":"2024-03-24T11:33:23.529914+0000","last_peered":"2024-03-24T11:33:23.529914+0000","last_clean":"2024-03-24T11:33:23.529914+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:33:23.529914+0000","last_undegraded":"2024-03-24T11:33:23.529914+0000","last_fullsized":"2024-03-24T11:33:23.529914+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:26.676209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:31:26.676209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"277'2906","reported_seq":2936,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.553507+0000","last_change":"2024-03-24T11:32:52.671396+0000","last_active":"2024-03-24T11:33:23.553507+0000","last_peered":"2024-03-24T11:33:23.553507+0000","last_clean":"2024-03-24T11:33:23.553507+0000","last_became_active":"2024-03-24T11:32:52.671052+0000","last_became_peered":"2024-03-24T11:32:52.671052+0000","last_unstale":"2024-03-24T11:33:23.553507+0000","last_undegraded":"2024-03-24T11:33:23.553507+0000","last_fullsized":"2024-03-24T11:33:23.553507+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:08.990427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190447616,"num_objects":2906,"num_object_clones":0,"num_object_copies":8718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2906,"num_write_kb":185984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":495,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.531228+0000","last_change":"2024-03-24T11:31:31.994930+0000","last_active":"2024-03-24T11:33:23.531228+0000","last_peered":"2024-03-24T11:33:23.531228+0000","last_clean":"2024-03-24T11:33:23.531228+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:33:23.531228+0000","last_undegraded":"2024-03-24T11:33:23.531228+0000","last_fullsized":"2024-03-24T11:33:23.531228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:31.994604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.994604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"213'17","reported_seq":507,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.531152+0000","last_change":"2024-03-24T11:32:38.137085+0000","last_active":"2024-03-24T11:33:23.531152+0000","last_peered":"2024-03-24T11:33:23.531152+0000","last_clean":"2024-03-24T11:33:23.531152+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:33:23.531152+0000","last_undegraded":"2024-03-24T11:33:23.531152+0000","last_fullsized":"2024-03-24T11:33:23.531152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'17","last_scrub_stamp":"2024-03-24T11:32:38.136966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.136966+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.504065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.532207+0000","last_change":"2024-03-24T11:32:18.855130+0000","last_active":"2024-03-24T11:33:23.532207+0000","last_peered":"2024-03-24T11:33:23.532207+0000","last_clean":"2024-03-24T11:33:23.532207+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:33:23.532207+0000","last_undegraded":"2024-03-24T11:33:23.532207+0000","last_fullsized":"2024-03-24T11:33:23.532207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:18.854979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.854979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":504,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.523876+0000","last_change":"2024-03-24T11:31:29.443192+0000","last_active":"2024-03-24T11:33:23.523876+0000","last_peered":"2024-03-24T11:33:23.523876+0000","last_clean":"2024-03-24T11:33:23.523876+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:33:23.523876+0000","last_undegraded":"2024-03-24T11:33:23.523876+0000","last_fullsized":"2024-03-24T11:33:23.523876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.443013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.443013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":105,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482561+0000","last_change":"2024-03-24T11:32:42.994801+0000","last_active":"2024-03-24T11:33:16.482561+0000","last_peered":"2024-03-24T11:33:16.482561+0000","last_clean":"2024-03-24T11:33:16.482561+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:33:16.482561+0000","last_undegraded":"2024-03-24T11:33:16.482561+0000","last_fullsized":"2024-03-24T11:33:16.482561+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.994694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.994694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.156201+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":405,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.234607+0000","last_change":"2024-03-24T11:33:20.172450+0000","last_active":"2024-03-24T11:33:24.234607+0000","last_peered":"2024-03-24T11:33:24.234607+0000","last_clean":"2024-03-24T11:33:24.234607+0000","last_became_active":"2024-03-24T11:30:35.356166+0000","last_became_peered":"2024-03-24T11:30:35.356166+0000","last_unstale":"2024-03-24T11:33:24.234607+0000","last_undegraded":"2024-03-24T11:33:24.234607+0000","last_fullsized":"2024-03-24T11:33:24.234607+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:20.172032+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.172032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:41.569472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"276'2824","reported_seq":2852,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.309808+0000","last_change":"2024-03-24T11:32:52.326526+0000","last_active":"2024-03-24T11:33:23.309808+0000","last_peered":"2024-03-24T11:33:23.309808+0000","last_clean":"2024-03-24T11:33:23.309808+0000","last_became_active":"2024-03-24T11:32:52.326215+0000","last_became_peered":"2024-03-24T11:32:52.326215+0000","last_unstale":"2024-03-24T11:33:23.309808+0000","last_undegraded":"2024-03-24T11:33:23.309808+0000","last_fullsized":"2024-03-24T11:33:23.309808+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.719177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185073664,"num_objects":2824,"num_object_clones":0,"num_object_copies":8472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":180736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":481,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.231575+0000","last_change":"2024-03-24T11:31:28.613941+0000","last_active":"2024-03-24T11:33:24.231575+0000","last_peered":"2024-03-24T11:33:24.231575+0000","last_clean":"2024-03-24T11:33:24.231575+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:33:24.231575+0000","last_undegraded":"2024-03-24T11:33:24.231575+0000","last_fullsized":"2024-03-24T11:33:24.231575+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.613752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.613752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"213'40","reported_seq":740,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525015+0000","last_change":"2024-03-24T11:32:43.782374+0000","last_active":"2024-03-24T11:33:23.525015+0000","last_peered":"2024-03-24T11:33:23.525015+0000","last_clean":"2024-03-24T11:33:23.525015+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:33:23.525015+0000","last_undegraded":"2024-03-24T11:33:23.525015+0000","last_fullsized":"2024-03-24T11:33:23.525015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'40","last_scrub_stamp":"2024-03-24T11:32:43.782256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.782256+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:01.406173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":485,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479602+0000","last_change":"2024-03-24T11:31:05.775644+0000","last_active":"2024-03-24T11:33:16.479602+0000","last_peered":"2024-03-24T11:33:16.479602+0000","last_clean":"2024-03-24T11:33:16.479602+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:33:16.479602+0000","last_undegraded":"2024-03-24T11:33:16.479602+0000","last_fullsized":"2024-03-24T11:33:16.479602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:05.775534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:05.775534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640021+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":506,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528068+0000","last_change":"2024-03-24T11:32:28.707278+0000","last_active":"2024-03-24T11:33:23.528068+0000","last_peered":"2024-03-24T11:33:23.528068+0000","last_clean":"2024-03-24T11:33:23.528068+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:33:23.528068+0000","last_undegraded":"2024-03-24T11:33:23.528068+0000","last_fullsized":"2024-03-24T11:33:23.528068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:32:28.707184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.707184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:33.691220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":461,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236971+0000","last_change":"2024-03-24T11:29:18.915411+0000","last_active":"2024-03-24T11:33:24.236971+0000","last_peered":"2024-03-24T11:33:24.236971+0000","last_clean":"2024-03-24T11:33:24.236971+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:33:24.236971+0000","last_undegraded":"2024-03-24T11:33:24.236971+0000","last_fullsized":"2024-03-24T11:33:24.236971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:18.915289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:29:18.915289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":642,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.532879+0000","last_change":"2024-03-24T11:32:29.691475+0000","last_active":"2024-03-24T11:33:23.532879+0000","last_peered":"2024-03-24T11:33:23.532879+0000","last_clean":"2024-03-24T11:33:23.532879+0000","last_became_active":"2024-03-24T11:28:38.173832+0000","last_became_peered":"2024-03-24T11:28:38.173832+0000","last_unstale":"2024-03-24T11:33:23.532879+0000","last_undegraded":"2024-03-24T11:33:23.532879+0000","last_fullsized":"2024-03-24T11:33:23.532879+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:29.691383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:29.691383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.888681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"277'2864","reported_seq":2894,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.663838+0000","last_change":"2024-03-24T11:32:52.671933+0000","last_active":"2024-03-24T11:33:23.663838+0000","last_peered":"2024-03-24T11:33:23.663838+0000","last_clean":"2024-03-24T11:33:23.663838+0000","last_became_active":"2024-03-24T11:32:52.671650+0000","last_became_peered":"2024-03-24T11:32:52.671650+0000","last_unstale":"2024-03-24T11:33:23.663838+0000","last_undegraded":"2024-03-24T11:33:23.663838+0000","last_fullsized":"2024-03-24T11:33:23.663838+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2864,"log_dups_size":0,"ondisk_log_size":2864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.692500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187695104,"num_objects":2864,"num_object_clones":0,"num_object_copies":8592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":183296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"213'15","reported_seq":518,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.235558+0000","last_change":"2024-03-24T11:32:37.060211+0000","last_active":"2024-03-24T11:33:24.235558+0000","last_peered":"2024-03-24T11:33:24.235558+0000","last_clean":"2024-03-24T11:33:24.235558+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:33:24.235558+0000","last_undegraded":"2024-03-24T11:33:24.235558+0000","last_fullsized":"2024-03-24T11:33:24.235558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:32:37.060067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.060067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.865083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":501,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528259+0000","last_change":"2024-03-24T11:31:20.465180+0000","last_active":"2024-03-24T11:33:23.528259+0000","last_peered":"2024-03-24T11:33:23.528259+0000","last_clean":"2024-03-24T11:33:23.528259+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:33:23.528259+0000","last_undegraded":"2024-03-24T11:33:23.528259+0000","last_fullsized":"2024-03-24T11:33:23.528259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:20.465096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:20.465096+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":495,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530029+0000","last_change":"2024-03-24T11:32:27.943222+0000","last_active":"2024-03-24T11:33:23.530029+0000","last_peered":"2024-03-24T11:33:23.530029+0000","last_clean":"2024-03-24T11:33:23.530029+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:33:23.530029+0000","last_undegraded":"2024-03-24T11:33:23.530029+0000","last_fullsized":"2024-03-24T11:33:23.530029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.943114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.943114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:36.987411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":477,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.231103+0000","last_change":"2024-03-24T11:31:23.762948+0000","last_active":"2024-03-24T11:33:24.231103+0000","last_peered":"2024-03-24T11:33:24.231103+0000","last_clean":"2024-03-24T11:33:24.231103+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:33:24.231103+0000","last_undegraded":"2024-03-24T11:33:24.231103+0000","last_fullsized":"2024-03-24T11:33:24.231103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:23.762486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:23.762486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.484839+0000","last_change":"2024-03-24T11:31:33.687541+0000","last_active":"2024-03-24T11:33:16.484839+0000","last_peered":"2024-03-24T11:33:16.484839+0000","last_clean":"2024-03-24T11:33:16.484839+0000","last_became_active":"2024-03-24T11:28:43.192246+0000","last_became_peered":"2024-03-24T11:28:43.192246+0000","last_unstale":"2024-03-24T11:33:16.484839+0000","last_undegraded":"2024-03-24T11:33:16.484839+0000","last_fullsized":"2024-03-24T11:33:16.484839+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:33.687261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.687261+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"278'2910","reported_seq":2942,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.372129+0000","last_change":"2024-03-24T11:32:52.673412+0000","last_active":"2024-03-24T11:33:24.372129+0000","last_peered":"2024-03-24T11:33:24.372129+0000","last_clean":"2024-03-24T11:33:24.372129+0000","last_became_active":"2024-03-24T11:32:52.673121+0000","last_became_peered":"2024-03-24T11:32:52.673121+0000","last_unstale":"2024-03-24T11:33:24.372129+0000","last_undegraded":"2024-03-24T11:33:24.372129+0000","last_fullsized":"2024-03-24T11:33:24.372129+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2910,"log_dups_size":0,"ondisk_log_size":2910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:09.827162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190709760,"num_objects":2910,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":186240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"213'44","reported_seq":561,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482010+0000","last_change":"2024-03-24T11:31:21.903188+0000","last_active":"2024-03-24T11:33:16.482010+0000","last_peered":"2024-03-24T11:33:16.482010+0000","last_clean":"2024-03-24T11:33:16.482010+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:33:16.482010+0000","last_undegraded":"2024-03-24T11:33:16.482010+0000","last_fullsized":"2024-03-24T11:33:16.482010+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:31:21.903023+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:31:21.903023+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.529186+0000","last_change":"2024-03-24T11:32:38.827248+0000","last_active":"2024-03-24T11:33:23.529186+0000","last_peered":"2024-03-24T11:33:23.529186+0000","last_clean":"2024-03-24T11:33:23.529186+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:33:23.529186+0000","last_undegraded":"2024-03-24T11:33:23.529186+0000","last_fullsized":"2024-03-24T11:33:23.529186+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.827167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.827167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.576619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":495,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530557+0000","last_change":"2024-03-24T11:31:22.849153+0000","last_active":"2024-03-24T11:33:23.530557+0000","last_peered":"2024-03-24T11:33:23.530557+0000","last_clean":"2024-03-24T11:33:23.530557+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:33:23.530557+0000","last_undegraded":"2024-03-24T11:33:23.530557+0000","last_fullsized":"2024-03-24T11:33:23.530557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.849049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:48.839016+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.849049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":107,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.531511+0000","last_change":"2024-03-24T11:33:22.957967+0000","last_active":"2024-03-24T11:33:23.531511+0000","last_peered":"2024-03-24T11:33:23.531511+0000","last_clean":"2024-03-24T11:33:23.531511+0000","last_became_active":"2024-03-24T11:30:35.355210+0000","last_became_peered":"2024-03-24T11:30:35.355210+0000","last_unstale":"2024-03-24T11:33:23.531511+0000","last_undegraded":"2024-03-24T11:33:23.531511+0000","last_fullsized":"2024-03-24T11:33:23.531511+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:22.957812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.957812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:38.348467+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":653,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482042+0000","last_change":"2024-03-24T11:32:34.973136+0000","last_active":"2024-03-24T11:33:16.482042+0000","last_peered":"2024-03-24T11:33:16.482042+0000","last_clean":"2024-03-24T11:33:16.482042+0000","last_became_active":"2024-03-24T11:28:43.191874+0000","last_became_peered":"2024-03-24T11:28:43.191874+0000","last_unstale":"2024-03-24T11:33:16.482042+0000","last_undegraded":"2024-03-24T11:33:16.482042+0000","last_fullsized":"2024-03-24T11:33:16.482042+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.973008+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.990059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"276'2735","reported_seq":2763,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.311136+0000","last_change":"2024-03-24T11:32:52.673866+0000","last_active":"2024-03-24T11:33:23.311136+0000","last_peered":"2024-03-24T11:33:23.311136+0000","last_clean":"2024-03-24T11:33:23.311136+0000","last_became_active":"2024-03-24T11:32:52.673498+0000","last_became_peered":"2024-03-24T11:32:52.673498+0000","last_unstale":"2024-03-24T11:33:23.311136+0000","last_undegraded":"2024-03-24T11:33:23.311136+0000","last_fullsized":"2024-03-24T11:33:23.311136+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2735,"log_dups_size":0,"ondisk_log_size":2735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:15.554040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":8205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":175040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":503,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530524+0000","last_change":"2024-03-24T11:32:44.207912+0000","last_active":"2024-03-24T11:33:23.530524+0000","last_peered":"2024-03-24T11:33:23.530524+0000","last_clean":"2024-03-24T11:33:23.530524+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:33:23.530524+0000","last_undegraded":"2024-03-24T11:33:23.530524+0000","last_fullsized":"2024-03-24T11:33:23.530524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:44.207817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:44.207817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.132453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"213'18","reported_seq":513,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480954+0000","last_change":"2024-03-24T11:31:11.798229+0000","last_active":"2024-03-24T11:33:16.480954+0000","last_peered":"2024-03-24T11:33:16.480954+0000","last_clean":"2024-03-24T11:33:16.480954+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:33:16.480954+0000","last_undegraded":"2024-03-24T11:33:16.480954+0000","last_fullsized":"2024-03-24T11:33:16.480954+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.798089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.798089+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640078+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":97,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:20.771519+0000","last_change":"2024-03-24T11:33:20.771559+0000","last_active":"2024-03-24T11:33:20.771519+0000","last_peered":"2024-03-24T11:33:20.771519+0000","last_clean":"2024-03-24T11:33:20.771519+0000","last_became_active":"2024-03-24T11:30:35.356055+0000","last_became_peered":"2024-03-24T11:30:35.356055+0000","last_unstale":"2024-03-24T11:33:20.771519+0000","last_undegraded":"2024-03-24T11:33:20.771519+0000","last_fullsized":"2024-03-24T11:33:20.771519+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:20.771417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.771417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:40.688846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":461,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238091+0000","last_change":"2024-03-24T11:30:12.206577+0000","last_active":"2024-03-24T11:33:24.238091+0000","last_peered":"2024-03-24T11:33:24.238091+0000","last_clean":"2024-03-24T11:33:24.238091+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:33:24.238091+0000","last_undegraded":"2024-03-24T11:33:24.238091+0000","last_fullsized":"2024-03-24T11:33:24.238091+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:12.206287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:12.206287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":641,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238256+0000","last_change":"2024-03-24T11:33:21.194045+0000","last_active":"2024-03-24T11:33:24.238256+0000","last_peered":"2024-03-24T11:33:24.238256+0000","last_clean":"2024-03-24T11:33:24.238256+0000","last_became_active":"2024-03-24T11:28:43.189494+0000","last_became_peered":"2024-03-24T11:28:43.189494+0000","last_unstale":"2024-03-24T11:33:24.238256+0000","last_undegraded":"2024-03-24T11:33:24.238256+0000","last_fullsized":"2024-03-24T11:33:24.238256+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:21.193899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:21.193899+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:25.212189+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"277'2814","reported_seq":2844,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.538844+0000","last_change":"2024-03-24T11:32:52.327253+0000","last_active":"2024-03-24T11:33:23.538844+0000","last_peered":"2024-03-24T11:33:23.538844+0000","last_clean":"2024-03-24T11:33:23.538844+0000","last_became_active":"2024-03-24T11:32:52.326946+0000","last_became_peered":"2024-03-24T11:32:52.326946+0000","last_unstale":"2024-03-24T11:33:23.538844+0000","last_undegraded":"2024-03-24T11:33:23.538844+0000","last_fullsized":"2024-03-24T11:33:23.538844+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":2814,"log_dups_size":0,"ondisk_log_size":2814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.621158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":180096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"238'9","reported_seq":164,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.481690+0000","last_change":"2024-03-24T11:32:43.964046+0000","last_active":"2024-03-24T11:33:16.481690+0000","last_peered":"2024-03-24T11:33:16.481690+0000","last_clean":"2024-03-24T11:33:16.481690+0000","last_became_active":"2024-03-24T11:30:35.359635+0000","last_became_peered":"2024-03-24T11:30:35.359635+0000","last_unstale":"2024-03-24T11:33:16.481690+0000","last_undegraded":"2024-03-24T11:33:16.481690+0000","last_fullsized":"2024-03-24T11:33:16.481690+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'9","last_scrub_stamp":"2024-03-24T11:32:43.963946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.963946+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.444511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"213'34","reported_seq":541,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.527370+0000","last_change":"2024-03-24T11:31:12.435804+0000","last_active":"2024-03-24T11:33:23.527370+0000","last_peered":"2024-03-24T11:33:23.527370+0000","last_clean":"2024-03-24T11:33:23.527370+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:33:23.527370+0000","last_undegraded":"2024-03-24T11:33:23.527370+0000","last_fullsized":"2024-03-24T11:33:23.527370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:31:12.435639+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:31:12.435639+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":299,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:22.820313+0000","last_change":"2024-03-24T11:33:22.820364+0000","last_active":"2024-03-24T11:33:22.820313+0000","last_peered":"2024-03-24T11:33:22.820313+0000","last_clean":"2024-03-24T11:33:22.820313+0000","last_became_active":"2024-03-24T11:30:35.356100+0000","last_became_peered":"2024-03-24T11:30:35.356100+0000","last_unstale":"2024-03-24T11:33:22.820313+0000","last_undegraded":"2024-03-24T11:33:22.820313+0000","last_fullsized":"2024-03-24T11:33:22.820313+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:33:22.820253+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.820253+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:44.928606+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":483,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528189+0000","last_change":"2024-03-24T11:31:14.458586+0000","last_active":"2024-03-24T11:33:23.528189+0000","last_peered":"2024-03-24T11:33:23.528189+0000","last_clean":"2024-03-24T11:33:23.528189+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:33:23.528189+0000","last_undegraded":"2024-03-24T11:33:23.528189+0000","last_fullsized":"2024-03-24T11:33:23.528189+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.458470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.458470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":556,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.484236+0000","last_change":"2024-03-24T11:31:04.735517+0000","last_active":"2024-03-24T11:33:16.484236+0000","last_peered":"2024-03-24T11:33:16.484236+0000","last_clean":"2024-03-24T11:33:16.484236+0000","last_became_active":"2024-03-24T11:30:35.354637+0000","last_became_peered":"2024-03-24T11:30:35.354637+0000","last_unstale":"2024-03-24T11:33:16.484236+0000","last_undegraded":"2024-03-24T11:33:16.484236+0000","last_fullsized":"2024-03-24T11:33:16.484236+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:04.735433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:04.735433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"213'15","reported_seq":490,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238003+0000","last_change":"2024-03-24T11:30:14.150582+0000","last_active":"2024-03-24T11:33:24.238003+0000","last_peered":"2024-03-24T11:33:24.238003+0000","last_clean":"2024-03-24T11:33:24.238003+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:33:24.238003+0000","last_undegraded":"2024-03-24T11:33:24.238003+0000","last_fullsized":"2024-03-24T11:33:24.238003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:14.150454+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.150454+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":509,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482476+0000","last_change":"2024-03-24T11:32:40.958471+0000","last_active":"2024-03-24T11:33:16.482476+0000","last_peered":"2024-03-24T11:33:16.482476+0000","last_clean":"2024-03-24T11:33:16.482476+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:33:16.482476+0000","last_undegraded":"2024-03-24T11:33:16.482476+0000","last_fullsized":"2024-03-24T11:33:16.482476+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:40.958389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.958389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:48.190213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":486,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.532404+0000","last_change":"2024-03-24T11:32:39.835187+0000","last_active":"2024-03-24T11:33:23.532404+0000","last_peered":"2024-03-24T11:33:23.532404+0000","last_clean":"2024-03-24T11:33:23.532404+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:33:23.532404+0000","last_undegraded":"2024-03-24T11:33:23.532404+0000","last_fullsized":"2024-03-24T11:33:23.532404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.835097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.835097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:03.202024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":105,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482659+0000","last_change":"2024-03-24T11:32:32.920081+0000","last_active":"2024-03-24T11:33:16.482659+0000","last_peered":"2024-03-24T11:33:16.482659+0000","last_clean":"2024-03-24T11:33:16.482659+0000","last_became_active":"2024-03-24T11:30:35.353212+0000","last_became_peered":"2024-03-24T11:30:35.353212+0000","last_unstale":"2024-03-24T11:33:16.482659+0000","last_undegraded":"2024-03-24T11:33:16.482659+0000","last_fullsized":"2024-03-24T11:33:16.482659+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.919963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:34.622437+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":473,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.527814+0000","last_change":"2024-03-24T11:31:24.533967+0000","last_active":"2024-03-24T11:33:23.527814+0000","last_peered":"2024-03-24T11:33:23.527814+0000","last_clean":"2024-03-24T11:33:23.527814+0000","last_became_active":"2024-03-24T11:28:43.189410+0000","last_became_peered":"2024-03-24T11:28:43.189410+0000","last_unstale":"2024-03-24T11:33:23.527814+0000","last_undegraded":"2024-03-24T11:33:23.527814+0000","last_fullsized":"2024-03-24T11:33:23.527814+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_clean_scrub_stamp":"2024-03-24T11:31:24.533874+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":649,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.524208+0000","last_change":"2024-03-24T11:32:33.790867+0000","last_active":"2024-03-24T11:33:23.524208+0000","last_peered":"2024-03-24T11:33:23.524208+0000","last_clean":"2024-03-24T11:33:23.524208+0000","last_became_active":"2024-03-24T11:28:43.192449+0000","last_became_peered":"2024-03-24T11:28:43.192449+0000","last_unstale":"2024-03-24T11:33:23.524208+0000","last_undegraded":"2024-03-24T11:33:23.524208+0000","last_fullsized":"2024-03-24T11:33:23.524208+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.790777+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.025287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"213'19","reported_seq":521,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.232160+0000","last_change":"2024-03-24T11:32:32.023601+0000","last_active":"2024-03-24T11:33:24.232160+0000","last_peered":"2024-03-24T11:33:24.232160+0000","last_clean":"2024-03-24T11:33:24.232160+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:33:24.232160+0000","last_undegraded":"2024-03-24T11:33:24.232160+0000","last_fullsized":"2024-03-24T11:33:24.232160+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:32.023502+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.023502+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.100674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":89,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480861+0000","last_change":"2024-03-24T11:30:35.359322+0000","last_active":"2024-03-24T11:33:16.480861+0000","last_peered":"2024-03-24T11:33:16.480861+0000","last_clean":"2024-03-24T11:33:16.480861+0000","last_became_active":"2024-03-24T11:30:35.358928+0000","last_became_peered":"2024-03-24T11:30:35.358928+0000","last_unstale":"2024-03-24T11:33:16.480861+0000","last_undegraded":"2024-03-24T11:33:16.480861+0000","last_fullsized":"2024-03-24T11:33:16.480861+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:21.749544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:21.749544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.639873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":464,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237861+0000","last_change":"2024-03-24T11:29:26.014447+0000","last_active":"2024-03-24T11:33:24.237861+0000","last_peered":"2024-03-24T11:33:24.237861+0000","last_clean":"2024-03-24T11:33:24.237861+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:33:24.237861+0000","last_undegraded":"2024-03-24T11:33:24.237861+0000","last_fullsized":"2024-03-24T11:33:24.237861+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:26.014199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:29:26.014199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":491,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530532+0000","last_change":"2024-03-24T11:32:26.898659+0000","last_active":"2024-03-24T11:33:23.530532+0000","last_peered":"2024-03-24T11:33:23.530532+0000","last_clean":"2024-03-24T11:33:23.530532+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:33:23.530532+0000","last_undegraded":"2024-03-24T11:33:23.530532+0000","last_fullsized":"2024-03-24T11:33:23.530532+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:26.898583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.898583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.612791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"213'15","reported_seq":516,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.529428+0000","last_change":"2024-03-24T11:31:27.541622+0000","last_active":"2024-03-24T11:33:23.529428+0000","last_peered":"2024-03-24T11:33:23.529428+0000","last_clean":"2024-03-24T11:33:23.529428+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:33:23.529428+0000","last_undegraded":"2024-03-24T11:33:23.529428+0000","last_fullsized":"2024-03-24T11:33:23.529428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:27.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.541510+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":497,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.234475+0000","last_change":"2024-03-24T11:32:45.985820+0000","last_active":"2024-03-24T11:33:24.234475+0000","last_peered":"2024-03-24T11:33:24.234475+0000","last_clean":"2024-03-24T11:33:24.234475+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:33:24.234475+0000","last_undegraded":"2024-03-24T11:33:24.234475+0000","last_fullsized":"2024-03-24T11:33:24.234475+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.985741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.985741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:14.491398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":656,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.239597+0000","last_change":"2024-03-24T11:31:13.201069+0000","last_active":"2024-03-24T11:33:24.239597+0000","last_peered":"2024-03-24T11:33:24.239597+0000","last_clean":"2024-03-24T11:33:24.239597+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:33:24.239597+0000","last_undegraded":"2024-03-24T11:33:24.239597+0000","last_fullsized":"2024-03-24T11:33:24.239597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:13.200503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.200503+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.486817+0000","last_change":"2024-03-24T11:32:27.764304+0000","last_active":"2024-03-24T11:33:16.486817+0000","last_peered":"2024-03-24T11:33:16.486817+0000","last_clean":"2024-03-24T11:33:16.486817+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:33:16.486817+0000","last_undegraded":"2024-03-24T11:33:16.486817+0000","last_fullsized":"2024-03-24T11:33:16.486817+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.764203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.764203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:55.837847+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":473,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.527759+0000","last_change":"2024-03-24T11:31:25.526652+0000","last_active":"2024-03-24T11:33:23.527759+0000","last_peered":"2024-03-24T11:33:23.527759+0000","last_clean":"2024-03-24T11:33:23.527759+0000","last_became_active":"2024-03-24T11:28:43.188941+0000","last_became_peered":"2024-03-24T11:28:43.188941+0000","last_unstale":"2024-03-24T11:33:23.527759+0000","last_undegraded":"2024-03-24T11:33:23.527759+0000","last_fullsized":"2024-03-24T11:33:23.527759+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.526522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.526522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"213'20","reported_seq":507,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238826+0000","last_change":"2024-03-24T11:33:19.230202+0000","last_active":"2024-03-24T11:33:24.238826+0000","last_peered":"2024-03-24T11:33:24.238826+0000","last_clean":"2024-03-24T11:33:24.238826+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:33:24.238826+0000","last_undegraded":"2024-03-24T11:33:24.238826+0000","last_fullsized":"2024-03-24T11:33:24.238826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:19.229920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.229920+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:47.588700+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":97,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480383+0000","last_change":"2024-03-24T11:31:30.728130+0000","last_active":"2024-03-24T11:33:16.480383+0000","last_peered":"2024-03-24T11:33:16.480383+0000","last_clean":"2024-03-24T11:33:16.480383+0000","last_became_active":"2024-03-24T11:30:35.353946+0000","last_became_peered":"2024-03-24T11:30:35.353946+0000","last_unstale":"2024-03-24T11:33:16.480383+0000","last_undegraded":"2024-03-24T11:33:16.480383+0000","last_fullsized":"2024-03-24T11:33:16.480383+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:30.728004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.728004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":668,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528458+0000","last_change":"2024-03-24T11:31:36.682644+0000","last_active":"2024-03-24T11:33:23.528458+0000","last_peered":"2024-03-24T11:33:23.528458+0000","last_clean":"2024-03-24T11:33:23.528458+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:33:23.528458+0000","last_undegraded":"2024-03-24T11:33:23.528458+0000","last_fullsized":"2024-03-24T11:33:23.528458+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:36.682542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.682542+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":476,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.232024+0000","last_change":"2024-03-24T11:32:31.071719+0000","last_active":"2024-03-24T11:33:24.232024+0000","last_peered":"2024-03-24T11:33:24.232024+0000","last_clean":"2024-03-24T11:33:24.232024+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:33:24.232024+0000","last_undegraded":"2024-03-24T11:33:24.232024+0000","last_fullsized":"2024-03-24T11:33:24.232024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:31.071621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:31.071621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:49.164745+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":489,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.229302+0000","last_change":"2024-03-24T11:31:06.835419+0000","last_active":"2024-03-24T11:33:24.229302+0000","last_peered":"2024-03-24T11:33:24.229302+0000","last_clean":"2024-03-24T11:33:24.229302+0000","last_became_active":"2024-03-24T11:28:43.188012+0000","last_became_peered":"2024-03-24T11:28:43.188012+0000","last_unstale":"2024-03-24T11:33:24.229302+0000","last_undegraded":"2024-03-24T11:33:24.229302+0000","last_fullsized":"2024-03-24T11:33:24.229302+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:06.835322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:06.835322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"213'24","reported_seq":525,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479958+0000","last_change":"2024-03-24T11:32:43.646863+0000","last_active":"2024-03-24T11:33:16.479958+0000","last_peered":"2024-03-24T11:33:16.479958+0000","last_clean":"2024-03-24T11:33:16.479958+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:33:16.479958+0000","last_undegraded":"2024-03-24T11:33:16.479958+0000","last_fullsized":"2024-03-24T11:33:16.479958+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:43.646773+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.646773+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:43.831542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":485,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479673+0000","last_change":"2024-03-24T11:31:37.636078+0000","last_active":"2024-03-24T11:33:16.479673+0000","last_peered":"2024-03-24T11:33:16.479673+0000","last_clean":"2024-03-24T11:33:16.479673+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:33:16.479673+0000","last_undegraded":"2024-03-24T11:33:16.479673+0000","last_fullsized":"2024-03-24T11:33:16.479673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:37.635705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.635705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640196+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":503,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525085+0000","last_change":"2024-03-24T11:32:36.752484+0000","last_active":"2024-03-24T11:33:23.525085+0000","last_peered":"2024-03-24T11:33:23.525085+0000","last_clean":"2024-03-24T11:33:23.525085+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:33:23.525085+0000","last_undegraded":"2024-03-24T11:33:23.525085+0000","last_fullsized":"2024-03-24T11:33:23.525085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.752406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.752406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:47.500544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":469,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236902+0000","last_change":"2024-03-24T11:32:32.077273+0000","last_active":"2024-03-24T11:33:24.236902+0000","last_peered":"2024-03-24T11:33:24.236902+0000","last_clean":"2024-03-24T11:33:24.236902+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:33:24.236902+0000","last_undegraded":"2024-03-24T11:33:24.236902+0000","last_fullsized":"2024-03-24T11:33:24.236902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.077150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.077150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:35.573524+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"213'15","reported_seq":516,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525287+0000","last_change":"2024-03-24T11:31:36.651472+0000","last_active":"2024-03-24T11:33:23.525287+0000","last_peered":"2024-03-24T11:33:23.525287+0000","last_clean":"2024-03-24T11:33:23.525287+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:33:23.525287+0000","last_undegraded":"2024-03-24T11:33:23.525287+0000","last_fullsized":"2024-03-24T11:33:23.525287+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:36.651340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.651340+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":101,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237641+0000","last_change":"2024-03-24T11:32:45.875224+0000","last_active":"2024-03-24T11:33:24.237641+0000","last_peered":"2024-03-24T11:33:24.237641+0000","last_clean":"2024-03-24T11:33:24.237641+0000","last_became_active":"2024-03-24T11:30:35.353230+0000","last_became_peered":"2024-03-24T11:30:35.353230+0000","last_unstale":"2024-03-24T11:33:24.237641+0000","last_undegraded":"2024-03-24T11:33:24.237641+0000","last_fullsized":"2024-03-24T11:33:24.237641+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.875145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.875145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:14.877519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":465,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237934+0000","last_change":"2024-03-24T11:29:29.048318+0000","last_active":"2024-03-24T11:33:24.237934+0000","last_peered":"2024-03-24T11:33:24.237934+0000","last_clean":"2024-03-24T11:33:24.237934+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:33:24.237934+0000","last_undegraded":"2024-03-24T11:33:24.237934+0000","last_fullsized":"2024-03-24T11:33:24.237934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:29:29.046315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:29:29.046315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":477,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.231842+0000","last_change":"2024-03-24T11:32:24.051941+0000","last_active":"2024-03-24T11:33:24.231842+0000","last_peered":"2024-03-24T11:33:24.231842+0000","last_clean":"2024-03-24T11:33:24.231842+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:33:24.231842+0000","last_undegraded":"2024-03-24T11:33:24.231842+0000","last_fullsized":"2024-03-24T11:33:24.231842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.051777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.051777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.050306+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":636,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530011+0000","last_change":"2024-03-24T11:32:40.219356+0000","last_active":"2024-03-24T11:33:23.530011+0000","last_peered":"2024-03-24T11:33:23.530011+0000","last_clean":"2024-03-24T11:33:23.530011+0000","last_became_active":"2024-03-24T11:28:43.192919+0000","last_became_peered":"2024-03-24T11:28:43.192919+0000","last_unstale":"2024-03-24T11:33:23.530011+0000","last_undegraded":"2024-03-24T11:33:23.530011+0000","last_fullsized":"2024-03-24T11:33:23.530011+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:40.219272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.219272+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:54.153521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"213'19","reported_seq":500,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237216+0000","last_change":"2024-03-24T11:30:13.139116+0000","last_active":"2024-03-24T11:33:24.237216+0000","last_peered":"2024-03-24T11:33:24.237216+0000","last_clean":"2024-03-24T11:33:24.237216+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:33:24.237216+0000","last_undegraded":"2024-03-24T11:33:24.237216+0000","last_fullsized":"2024-03-24T11:33:24.237216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:30:13.138980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:13.138980+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":494,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.532502+0000","last_change":"2024-03-24T11:31:40.668063+0000","last_active":"2024-03-24T11:33:23.532502+0000","last_peered":"2024-03-24T11:33:23.532502+0000","last_clean":"2024-03-24T11:33:23.532502+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:33:23.532502+0000","last_undegraded":"2024-03-24T11:33:23.532502+0000","last_fullsized":"2024-03-24T11:33:23.532502+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:40.667849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:40.667849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":97,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479985+0000","last_change":"2024-03-24T11:32:23.732096+0000","last_active":"2024-03-24T11:33:16.479985+0000","last_peered":"2024-03-24T11:33:16.479985+0000","last_clean":"2024-03-24T11:33:16.479985+0000","last_became_active":"2024-03-24T11:30:35.353571+0000","last_became_peered":"2024-03-24T11:30:35.353571+0000","last_unstale":"2024-03-24T11:33:16.479985+0000","last_undegraded":"2024-03-24T11:33:16.479985+0000","last_fullsized":"2024-03-24T11:33:16.479985+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:23.731967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.731967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:38.504397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":469,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238177+0000","last_change":"2024-03-24T11:30:24.313618+0000","last_active":"2024-03-24T11:33:24.238177+0000","last_peered":"2024-03-24T11:33:24.238177+0000","last_clean":"2024-03-24T11:33:24.238177+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:33:24.238177+0000","last_undegraded":"2024-03-24T11:33:24.238177+0000","last_fullsized":"2024-03-24T11:33:24.238177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:24.313152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.313152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"213'10","reported_seq":483,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236522+0000","last_change":"2024-03-24T11:29:30.032558+0000","last_active":"2024-03-24T11:33:24.236522+0000","last_peered":"2024-03-24T11:33:24.236522+0000","last_clean":"2024-03-24T11:33:24.236522+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:33:24.236522+0000","last_undegraded":"2024-03-24T11:33:24.236522+0000","last_fullsized":"2024-03-24T11:33:24.236522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:29:30.032476+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:29:30.032476+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":496,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.240078+0000","last_change":"2024-03-24T11:32:36.013375+0000","last_active":"2024-03-24T11:33:24.240078+0000","last_peered":"2024-03-24T11:33:24.240078+0000","last_clean":"2024-03-24T11:33:24.240078+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:33:24.240078+0000","last_undegraded":"2024-03-24T11:33:24.240078+0000","last_fullsized":"2024-03-24T11:33:24.240078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.013287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.013287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:44.208803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":501,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.481748+0000","last_change":"2024-03-24T11:32:38.991091+0000","last_active":"2024-03-24T11:33:16.481748+0000","last_peered":"2024-03-24T11:33:16.481748+0000","last_clean":"2024-03-24T11:33:16.481748+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:33:16.481748+0000","last_undegraded":"2024-03-24T11:33:16.481748+0000","last_fullsized":"2024-03-24T11:33:16.481748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.991009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.991009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:05.887138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":107,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530221+0000","last_change":"2024-03-24T11:32:43.213245+0000","last_active":"2024-03-24T11:33:23.530221+0000","last_peered":"2024-03-24T11:33:23.530221+0000","last_clean":"2024-03-24T11:33:23.530221+0000","last_became_active":"2024-03-24T11:30:35.353704+0000","last_became_peered":"2024-03-24T11:30:35.353704+0000","last_unstale":"2024-03-24T11:33:23.530221+0000","last_undegraded":"2024-03-24T11:33:23.530221+0000","last_fullsized":"2024-03-24T11:33:23.530221+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:43.213130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.213130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:06.338202+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"213'24","reported_seq":541,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482338+0000","last_change":"2024-03-24T11:32:30.897616+0000","last_active":"2024-03-24T11:33:16.482338+0000","last_peered":"2024-03-24T11:33:16.482338+0000","last_clean":"2024-03-24T11:33:16.482338+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:33:16.482338+0000","last_undegraded":"2024-03-24T11:33:16.482338+0000","last_fullsized":"2024-03-24T11:33:16.482338+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:30.897405+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.897405+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:53.068479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":480,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236436+0000","last_change":"2024-03-24T11:32:30.073961+0000","last_active":"2024-03-24T11:33:24.236436+0000","last_peered":"2024-03-24T11:33:24.236436+0000","last_clean":"2024-03-24T11:33:24.236436+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:33:24.236436+0000","last_undegraded":"2024-03-24T11:33:24.236436+0000","last_fullsized":"2024-03-24T11:33:24.236436+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:30.073873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.073873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:52.383142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":482,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.232259+0000","last_change":"2024-03-24T11:31:14.751432+0000","last_active":"2024-03-24T11:33:24.232259+0000","last_peered":"2024-03-24T11:33:24.232259+0000","last_clean":"2024-03-24T11:33:24.232259+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:33:24.232259+0000","last_undegraded":"2024-03-24T11:33:24.232259+0000","last_fullsized":"2024-03-24T11:33:24.232259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.751309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.751309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":490,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525996+0000","last_change":"2024-03-24T11:31:41.853862+0000","last_active":"2024-03-24T11:33:23.525996+0000","last_peered":"2024-03-24T11:33:23.525996+0000","last_clean":"2024-03-24T11:33:23.525996+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:33:23.525996+0000","last_undegraded":"2024-03-24T11:33:23.525996+0000","last_fullsized":"2024-03-24T11:33:23.525996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.853751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"213'20","reported_seq":524,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.232134+0000","last_change":"2024-03-24T11:32:25.904952+0000","last_active":"2024-03-24T11:33:24.232134+0000","last_peered":"2024-03-24T11:33:24.232134+0000","last_clean":"2024-03-24T11:33:24.232134+0000","last_became_active":"2024-03-24T11:18:56.141024+0000","last_became_peered":"2024-03-24T11:18:56.141024+0000","last_unstale":"2024-03-24T11:33:24.232134+0000","last_undegraded":"2024-03-24T11:33:24.232134+0000","last_fullsized":"2024-03-24T11:33:24.232134+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:25.904851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.904851+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:28.755461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":503,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.523863+0000","last_change":"2024-03-24T11:31:37.672115+0000","last_active":"2024-03-24T11:33:23.523863+0000","last_peered":"2024-03-24T11:33:23.523863+0000","last_clean":"2024-03-24T11:33:23.523863+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:33:23.523863+0000","last_undegraded":"2024-03-24T11:33:23.523863+0000","last_fullsized":"2024-03-24T11:33:23.523863+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:37.671989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.671989+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":690,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.482191+0000","last_change":"2024-03-24T11:32:41.963960+0000","last_active":"2024-03-24T11:33:16.482191+0000","last_peered":"2024-03-24T11:33:16.482191+0000","last_clean":"2024-03-24T11:33:16.482191+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:33:16.482191+0000","last_undegraded":"2024-03-24T11:33:16.482191+0000","last_fullsized":"2024-03-24T11:33:16.482191+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.963825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:50.606211+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":138,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528016+0000","last_change":"2024-03-24T11:30:35.355569+0000","last_active":"2024-03-24T11:33:23.528016+0000","last_peered":"2024-03-24T11:33:23.528016+0000","last_clean":"2024-03-24T11:33:23.528016+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:33:23.528016+0000","last_undegraded":"2024-03-24T11:33:23.528016+0000","last_fullsized":"2024-03-24T11:33:23.528016+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:30:22.784182+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:30:22.784182+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"213'18","reported_seq":500,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237172+0000","last_change":"2024-03-24T11:31:11.260142+0000","last_active":"2024-03-24T11:33:24.237172+0000","last_peered":"2024-03-24T11:33:24.237172+0000","last_clean":"2024-03-24T11:33:24.237172+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:33:24.237172+0000","last_undegraded":"2024-03-24T11:33:24.237172+0000","last_fullsized":"2024-03-24T11:33:24.237172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.259991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.259991+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":484,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:19.752866+0000","last_change":"2024-03-24T11:33:19.752899+0000","last_active":"2024-03-24T11:33:19.752866+0000","last_peered":"2024-03-24T11:33:19.752866+0000","last_clean":"2024-03-24T11:33:19.752866+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:33:19.752866+0000","last_undegraded":"2024-03-24T11:33:19.752866+0000","last_fullsized":"2024-03-24T11:33:19.752866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.752717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.752717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:25.705201+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":484,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.484533+0000","last_change":"2024-03-24T11:32:38.742586+0000","last_active":"2024-03-24T11:33:16.484533+0000","last_peered":"2024-03-24T11:33:16.484533+0000","last_clean":"2024-03-24T11:33:16.484533+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:33:16.484533+0000","last_undegraded":"2024-03-24T11:33:16.484533+0000","last_fullsized":"2024-03-24T11:33:16.484533+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.742478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.742478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:40.063731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":461,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237468+0000","last_change":"2024-03-24T11:30:08.308986+0000","last_active":"2024-03-24T11:33:24.237468+0000","last_peered":"2024-03-24T11:33:24.237468+0000","last_clean":"2024-03-24T11:33:24.237468+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:33:24.237468+0000","last_undegraded":"2024-03-24T11:33:24.237468+0000","last_fullsized":"2024-03-24T11:33:24.237468+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:08.308860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:08.308860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"213'27","reported_seq":541,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.530209+0000","last_change":"2024-03-24T11:32:35.070335+0000","last_active":"2024-03-24T11:33:23.530209+0000","last_peered":"2024-03-24T11:33:23.530209+0000","last_clean":"2024-03-24T11:33:23.530209+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:33:23.530209+0000","last_undegraded":"2024-03-24T11:33:23.530209+0000","last_fullsized":"2024-03-24T11:33:23.530209+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:32:35.070232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:35.070232+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:37.835251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":496,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.235422+0000","last_change":"2024-03-24T11:33:22.182976+0000","last_active":"2024-03-24T11:33:24.235422+0000","last_peered":"2024-03-24T11:33:24.235422+0000","last_clean":"2024-03-24T11:33:24.235422+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:33:24.235422+0000","last_undegraded":"2024-03-24T11:33:24.235422+0000","last_fullsized":"2024-03-24T11:33:24.235422+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:22.182859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.182859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:32.867301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":494,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.525049+0000","last_change":"2024-03-24T11:32:42.808822+0000","last_active":"2024-03-24T11:33:23.525049+0000","last_peered":"2024-03-24T11:33:23.525049+0000","last_clean":"2024-03-24T11:33:23.525049+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:33:23.525049+0000","last_undegraded":"2024-03-24T11:33:23.525049+0000","last_fullsized":"2024-03-24T11:33:23.525049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.808738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.808738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:59.742514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":474,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.485304+0000","last_change":"2024-03-24T11:31:09.866310+0000","last_active":"2024-03-24T11:33:16.485304+0000","last_peered":"2024-03-24T11:33:16.485304+0000","last_clean":"2024-03-24T11:33:16.485304+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:33:16.485304+0000","last_undegraded":"2024-03-24T11:33:16.485304+0000","last_fullsized":"2024-03-24T11:33:16.485304+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:09.866232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:31:09.866232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.639993+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"213'7","reported_seq":493,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479495+0000","last_change":"2024-03-24T11:31:27.652238+0000","last_active":"2024-03-24T11:33:16.479495+0000","last_peered":"2024-03-24T11:33:16.479495+0000","last_clean":"2024-03-24T11:33:16.479495+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:33:16.479495+0000","last_undegraded":"2024-03-24T11:33:16.479495+0000","last_fullsized":"2024-03-24T11:33:16.479495+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:31:27.652037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.652037+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640109+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":494,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528149+0000","last_change":"2024-03-24T11:33:19.374929+0000","last_active":"2024-03-24T11:33:23.528149+0000","last_peered":"2024-03-24T11:33:23.528149+0000","last_clean":"2024-03-24T11:33:23.528149+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:33:23.528149+0000","last_undegraded":"2024-03-24T11:33:23.528149+0000","last_fullsized":"2024-03-24T11:33:23.528149+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.374805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.374805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:37.994295+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":97,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480159+0000","last_change":"2024-03-24T11:31:41.715026+0000","last_active":"2024-03-24T11:33:16.480159+0000","last_peered":"2024-03-24T11:33:16.480159+0000","last_clean":"2024-03-24T11:33:16.480159+0000","last_became_active":"2024-03-24T11:30:35.359239+0000","last_became_peered":"2024-03-24T11:30:35.359239+0000","last_unstale":"2024-03-24T11:33:16.480159+0000","last_undegraded":"2024-03-24T11:33:16.480159+0000","last_fullsized":"2024-03-24T11:33:16.480159+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.640272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":474,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.532888+0000","last_change":"2024-03-24T11:30:17.465431+0000","last_active":"2024-03-24T11:33:23.532888+0000","last_peered":"2024-03-24T11:33:23.532888+0000","last_clean":"2024-03-24T11:33:23.532888+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:33:23.532888+0000","last_undegraded":"2024-03-24T11:33:23.532888+0000","last_fullsized":"2024-03-24T11:33:23.532888+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:17.465226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:17.465226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"213'15","reported_seq":497,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479652+0000","last_change":"2024-03-24T11:30:15.882584+0000","last_active":"2024-03-24T11:33:16.479652+0000","last_peered":"2024-03-24T11:33:16.479652+0000","last_clean":"2024-03-24T11:33:16.479652+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:33:16.479652+0000","last_undegraded":"2024-03-24T11:33:16.479652+0000","last_fullsized":"2024-03-24T11:33:16.479652+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:15.882446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:30:15.882446+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.639813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":494,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528392+0000","last_change":"2024-03-24T11:31:41.687584+0000","last_active":"2024-03-24T11:33:23.528392+0000","last_peered":"2024-03-24T11:33:23.528392+0000","last_clean":"2024-03-24T11:33:23.528392+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:33:23.528392+0000","last_undegraded":"2024-03-24T11:33:23.528392+0000","last_fullsized":"2024-03-24T11:33:23.528392+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.687474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.687474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":486,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.528311+0000","last_change":"2024-03-24T11:31:15.474018+0000","last_active":"2024-03-24T11:33:23.528311+0000","last_peered":"2024-03-24T11:33:23.528311+0000","last_clean":"2024-03-24T11:33:23.528311+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:33:23.528311+0000","last_undegraded":"2024-03-24T11:33:23.528311+0000","last_fullsized":"2024-03-24T11:33:23.528311+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:15.473909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:31:15.473909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":484,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.232223+0000","last_change":"2024-03-24T11:33:24.183098+0000","last_active":"2024-03-24T11:33:24.232223+0000","last_peered":"2024-03-24T11:33:24.232223+0000","last_clean":"2024-03-24T11:33:24.232223+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:33:24.232223+0000","last_undegraded":"2024-03-24T11:33:24.232223+0000","last_fullsized":"2024-03-24T11:33:24.232223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:24.182966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.182966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:34.665679+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"213'27","reported_seq":511,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.237263+0000","last_change":"2024-03-24T11:31:41.107651+0000","last_active":"2024-03-24T11:33:24.237263+0000","last_peered":"2024-03-24T11:33:24.237263+0000","last_clean":"2024-03-24T11:33:24.237263+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:33:24.237263+0000","last_undegraded":"2024-03-24T11:33:24.237263+0000","last_fullsized":"2024-03-24T11:33:24.237263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:31:41.107512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.107512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":474,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.236620+0000","last_change":"2024-03-24T11:30:23.137546+0000","last_active":"2024-03-24T11:33:24.236620+0000","last_peered":"2024-03-24T11:33:24.236620+0000","last_clean":"2024-03-24T11:33:24.236620+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:33:24.236620+0000","last_undegraded":"2024-03-24T11:33:24.236620+0000","last_fullsized":"2024-03-24T11:33:24.236620+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:23.137393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:23.137393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":476,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480167+0000","last_change":"2024-03-24T11:32:24.692923+0000","last_active":"2024-03-24T11:33:16.480167+0000","last_peered":"2024-03-24T11:33:16.480167+0000","last_clean":"2024-03-24T11:33:16.480167+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:33:16.480167+0000","last_undegraded":"2024-03-24T11:33:16.480167+0000","last_fullsized":"2024-03-24T11:33:16.480167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.692844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.692844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:31.656723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":460,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.238011+0000","last_change":"2024-03-24T11:31:14.327893+0000","last_active":"2024-03-24T11:33:24.238011+0000","last_peered":"2024-03-24T11:33:24.238011+0000","last_clean":"2024-03-24T11:33:24.238011+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:33:24.238011+0000","last_undegraded":"2024-03-24T11:33:24.238011+0000","last_fullsized":"2024-03-24T11:33:24.238011+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.327777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.327777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"213'15","reported_seq":521,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.480083+0000","last_change":"2024-03-24T11:31:08.802687+0000","last_active":"2024-03-24T11:33:16.480083+0000","last_peered":"2024-03-24T11:33:16.480083+0000","last_clean":"2024-03-24T11:33:16.480083+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:33:16.480083+0000","last_undegraded":"2024-03-24T11:33:16.480083+0000","last_fullsized":"2024-03-24T11:33:16.480083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:08.802538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.802538+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.639938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":489,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-24T11:33:24.239849+0000","last_change":"2024-03-24T11:32:35.031834+0000","last_active":"2024-03-24T11:33:24.239849+0000","last_peered":"2024-03-24T11:33:24.239849+0000","last_clean":"2024-03-24T11:33:24.239849+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:33:24.239849+0000","last_undegraded":"2024-03-24T11:33:24.239849+0000","last_fullsized":"2024-03-24T11:33:24.239849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:35.031735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:35.031735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:42.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":494,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-03-24T11:33:23.523771+0000","last_change":"2024-03-24T11:32:28.712422+0000","last_active":"2024-03-24T11:33:23.523771+0000","last_peered":"2024-03-24T11:33:23.523771+0000","last_clean":"2024-03-24T11:33:23.523771+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:33:23.523771+0000","last_undegraded":"2024-03-24T11:33:23.523771+0000","last_fullsized":"2024-03-24T11:33:23.523771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.712294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.712294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:51.891340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-24T11:33:16.479389+0000","last_change":"2024-03-24T11:31:07.885306+0000","last_active":"2024-03-24T11:33:16.479389+0000","last_peered":"2024-03-24T11:33:16.479389+0000","last_clean":"2024-03-24T11:33:16.479389+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:33:16.479389+0000","last_undegraded":"2024-03-24T11:33:16.479389+0000","last_fullsized":"2024-03-24T11:33:16.479389+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:07.885177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:07.885177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:17.639903+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2968059904,"num_objects":45289,"num_object_clones":0,"num_object_copies":135867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45289,"num_write_kb":2898496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8964145152,"data_stored":8964145152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45289,"ondisk_log_size":45289,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":175,"num_read_kb":170,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070642,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1187300,"kb_used_data":704924,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92557852,"statfs":{"total":95995035648,"available":94779240448,"internally_reserved":0,"allocated":721842176,"data_stored":721496068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":493943243},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364983,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1586200,"kb_used_data":1234320,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92158952,"statfs":{"total":95995035648,"available":94370766848,"internally_reserved":0,"allocated":1263943680,"data_stored":1263610081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1047680,"kb_used_data":533368,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92697472,"statfs":{"total":95995035648,"available":94922211328,"internally_reserved":0,"allocated":546168832,"data_stored":545795110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738582,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2344508,"kb_used_data":1848240,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91400644,"statfs":{"total":95995035648,"available":93594259456,"internally_reserved":0,"allocated":1892597760,"data_stored":1892239144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411532,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458404,"kb_used_data":1075996,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92286748,"statfs":{"total":95995035648,"available":94501629952,"internally_reserved":0,"allocated":1101819904,"data_stored":1101473856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149893,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1429084,"kb_used_data":928656,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92316068,"statfs":{"total":95995035648,"available":94531653632,"internally_reserved":0,"allocated":950943744,"data_stored":950561566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":512424391},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986369,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1764848,"kb_used_data":1400488,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91980304,"statfs":{"total":95995035648,"available":94187831296,"internally_reserved":0,"allocated":1434099712,"data_stored":1433741744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659324,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418912,"kb_used_data":904600,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92326240,"statfs":{"total":95995035648,"available":94542069760,"internally_reserved":0,"allocated":926310400,"data_stored":925968714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1908801536,"data_stored":1908801536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":563740672,"data_stored":563740672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1130233856,"data_stored":1130233856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":949944320,"data_stored":949944320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1458962432,"data_stored":1458962432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":930086912,"data_stored":930086912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1284177920,"data_stored":1284177920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":738197504,"data_stored":738197504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:33:28.370 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T11:33:28.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-03-24T11:33:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:28.372 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:28.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:28.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:28.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:28.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:28.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:28.623 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 25 15 53370 53355 133.367 101.438 0.0423655 0.00749208 2024-03-24T11:33:28.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:28.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[29608]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:33:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:33:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:33:28.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2816353893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[29608]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[29608]: 6.19 deep-scrub starts 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[37907]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2816353893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[37907]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T11:33:28.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:28 smithi179 ceph-mon[37907]: 6.19 deep-scrub starts 2024-03-24T11:33:28.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:28.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:28.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:28.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:28.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:28 smithi181 ceph-mon[33998]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:33:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:28 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:33:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:28 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:33:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2816353893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:33:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:28 smithi181 ceph-mon[33998]: osdmap e279: 8 total, 8 up, 8 in 2024-03-24T11:33:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:28 smithi181 ceph-mon[33998]: 6.19 deep-scrub starts 2024-03-24T11:33:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:28.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:28.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:29.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:29.068 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:29.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:29.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:29.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:29.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:29.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:29.348 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:29.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:29.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:29.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:29.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:29.587 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:29.623 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 26 16 55025 55009 132.214 103.375 0.00867243 0.00755888 2024-03-24T11:33:29.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:29.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:29.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:29.765 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:29.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:29.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:29.894 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: 5.b deep-scrub starts 2024-03-24T11:33:29.894 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: 5.b deep-scrub ok 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: pgmap v617: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: 6.19 deep-scrub ok 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[29608]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: 5.b deep-scrub starts 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: 5.b deep-scrub ok 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:29.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:33:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:33:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: pgmap v617: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-24T11:33:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T11:33:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: 6.19 deep-scrub ok 2024-03-24T11:33:29.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:29 smithi179 ceph-mon[37907]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T11:33:29.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: 5.b deep-scrub starts 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: 5.b deep-scrub ok 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: pgmap v617: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: osdmap e280: 8 total, 8 up, 8 in 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: 6.19 deep-scrub ok 2024-03-24T11:33:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:29 smithi181 ceph-mon[33998]: osdmap e281: 8 total, 8 up, 8 in 2024-03-24T11:33:29.908 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:29.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:29.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:30.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:30.044 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:30.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:30.146 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:30.184 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:30.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:30.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:30.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:30.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:30.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:30.459 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:30.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:30.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:30.598 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:30.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:30.623 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 27 16 56629 56613 131.03 100.25 0.00832959 0.0076269 2024-03-24T11:33:30.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:30.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:30.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:30.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:30.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:30 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:33:30.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:30 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:33:30.808 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:30 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:33:30.808 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:30 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:33:30.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:30.870 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:30.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:30.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:30 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:33:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:30 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:33:30.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:31.016 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:31.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:31.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:31.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:31.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:31.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:33:31.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:31.290 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:31.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:31.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:31.430 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:31.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:31.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:31.592 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:31.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:31.623 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 28 16 58082 58066 129.593 90.8125 0.0125466 0.00771186 2024-03-24T11:33:31.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:31.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:31.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:31.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:31.871 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:31.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:31.873 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:33:31.873 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:33:31.873 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:33:31.873 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:33:31.873 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:33:31.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[29608]: 1.11 scrub starts 2024-03-24T11:33:31.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[29608]: 1.11 scrub ok 2024-03-24T11:33:31.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[29608]: pgmap v620: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[29608]: 6.2 scrub starts 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[29608]: 6.2 scrub ok 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[37907]: 1.11 scrub starts 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[37907]: 1.11 scrub ok 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[37907]: pgmap v620: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[37907]: 6.2 scrub starts 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[37907]: 6.2 scrub ok 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:33:31.874 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:31 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:33:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:31 smithi181 ceph-mon[33998]: 1.11 scrub starts 2024-03-24T11:33:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:31 smithi181 ceph-mon[33998]: 1.11 scrub ok 2024-03-24T11:33:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:31 smithi181 ceph-mon[33998]: pgmap v620: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-24T11:33:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:31 smithi181 ceph-mon[33998]: 6.2 scrub starts 2024-03-24T11:33:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:31 smithi181 ceph-mon[33998]: 6.2 scrub ok 2024-03-24T11:33:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:31 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:33:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:31 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:33:31.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:32.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:32.105 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:33:32.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:32.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:33:32.160 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:32.189 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:33:32.230 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:33:32.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:32.271 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:33:32.305 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:32.311 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:33:32.351 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:33:32.393 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:33:32.400 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:33:32.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:32.449 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:32.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:32.589 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:32.623 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 29 16 59502 59486 128.184 88.75 0.00505791 0.00779675 2024-03-24T11:33:32.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:32.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:32 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3682121142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:32.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]: dispatch 2024-03-24T11:33:32.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]: dispatch 2024-03-24T11:33:32.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:32 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3682121142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:32.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]: dispatch 2024-03-24T11:33:32.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]: dispatch 2024-03-24T11:33:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:32.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:32.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:33:32.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:32.860 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:32 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3682121142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]: dispatch 2024-03-24T11:33:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]: dispatch 2024-03-24T11:33:32.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:32.996 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:32.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:33.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:33.149 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:33.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:33.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:33.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:33.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:33.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:33.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:33.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:33.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:33.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:33.610 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:33.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 30 16 60942 60926 126.907 90 0.0102775 0.00787388 2024-03-24T11:33:33.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:33.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:33.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:33.717 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: pgmap v621: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: 5.17 scrub starts 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: 5.17 scrub ok 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]': finished 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: 1.6 deep-scrub starts 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: 1.6 deep-scrub ok 2024-03-24T11:33:33.771 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: pgmap v621: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: 5.17 scrub starts 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: 5.17 scrub ok 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]': finished 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: 1.6 deep-scrub starts 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: 1.6 deep-scrub ok 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:33:33.772 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:33.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:33.831 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:33.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:33.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:33.972 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:33.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:34.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:34.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: pgmap v621: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: 5.17 scrub starts 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: 5.17 scrub ok 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 2]}]': finished 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: osdmap e282: 8 total, 8 up, 8 in 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: 1.6 deep-scrub starts 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: 1.6 deep-scrub ok 2024-03-24T11:33:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:33:34.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:34.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:33:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:33:34.213 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:34.255 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:34.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:34.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:34.404 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:34.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:34.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:34.544 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:34.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:34.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 31 16 62392 62376 125.736 90.625 0.0134033 0.00794862 2024-03-24T11:33:34.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:34.688 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:34.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:34.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:34.828 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:34.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 4.10 scrub starts 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 4.10 scrub ok 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:33:34.867 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 4.10 scrub starts 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 4.10 scrub ok 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:34.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:33:34.869 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:34 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:33:34.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:34.965 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:34.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:35.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:35.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:35.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:35.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 4.10 scrub starts 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 4.10 scrub ok 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: osdmap e283: 8 total, 8 up, 8 in 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:33:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:34 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:33:35.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:35.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:33:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:33:35.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:35.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:35.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:35.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:35.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:35.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:35.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:35.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:35.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:33:35.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:35.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 32 16 63903 63887 124.758 94.4375 0.0151586 0.00801159 2024-03-24T11:33:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:35.656 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:35.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:35.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:35.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:35.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:35.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:35.937 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:35.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:33:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:33:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[29608]: pgmap v624: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:33:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:33:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:33:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:33:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:33:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[37907]: pgmap v624: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:33:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:33:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:35 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:33:36.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:36.069 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:36.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:35 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:33:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:35 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:33:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:35 smithi181 ceph-mon[33998]: pgmap v624: 168 pgs: 168 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-24T11:33:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:35 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:33:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:35 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:33:36.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:36.227 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:36.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:36.329 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:36.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:36.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:36.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:36.517 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:36.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:36.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:36.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 33 16 65247 65231 123.523 84 0.0107815 0.00809106 2024-03-24T11:33:36.649 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:36.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:36.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:36.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:36.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:36.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:36.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:36.929 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:36.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:33:36.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:33:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:33:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:33:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:33:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:36 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:33:37.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:37.057 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:37.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:36 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:33:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:36 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:33:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:36 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:33:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:36 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:33:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:36 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:33:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:36 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:33:37.158 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:37.191 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:37.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:37.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:37.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:37.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:37.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:37.455 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:37.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:37.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:37.595 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:37.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:37.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 34 16 66559 66543 122.301 82 0.0124835 0.00817167 2024-03-24T11:33:37.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:37.740 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:37.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:37.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:37.887 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:37.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:37.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: pgmap v625: 168 pgs: 168 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 721 B/s rd, 75 MiB/s wr, 1.20k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: 3.7 deep-scrub starts 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: 3.7 deep-scrub ok 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: pgmap v625: 168 pgs: 168 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 721 B/s rd, 75 MiB/s wr, 1.20k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: 3.7 deep-scrub starts 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: 3.7 deep-scrub ok 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:33:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:33:37.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:37.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:37.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:33:37.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:37 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:33:37.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:38.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:38.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:38.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: pgmap v625: 168 pgs: 168 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 721 B/s rd, 75 MiB/s wr, 1.20k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: 3.7 deep-scrub starts 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: 3.7 deep-scrub ok 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:37 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:33:38.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:38.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:38.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:38.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:38.442 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:38.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.581 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:38.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:38.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 35 16 67889 67873 121.182 83.125 0.00518621 0.00824708 2024-03-24T11:33:38.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:38.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:38.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:38.856 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:38.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:33:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:33:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[29608]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T11:33:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:33:38.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:33:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:33:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:33:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[37907]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T11:33:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:33:38.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:38 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:33:38.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:38.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:39.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:39.131 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:39.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:39.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:38 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:33:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:38 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:33:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:38 smithi181 ceph-mon[33998]: osdmap e284: 8 total, 8 up, 8 in 2024-03-24T11:33:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:38 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:33:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:38 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:33:39.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:39.272 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:39.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:39.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:39.411 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:39.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:39.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:39.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:39.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:39.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 36 16 69141 69125 119.989 78.25 0.00311329 0.00830068 2024-03-24T11:33:39.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:39.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:39.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:39.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:39.810 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:39.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:39.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:39.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:39.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:39.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: pgmap v627: 168 pgs: 168 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.48k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:39.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: 6.c scrub starts 2024-03-24T11:33:39.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: 6.c scrub ok 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[29608]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: pgmap v627: 168 pgs: 168 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.48k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: 6.c scrub starts 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: 6.c scrub ok 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:33:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:33:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:39 smithi179 ceph-mon[37907]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T11:33:40.037 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:40.075 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:40.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: pgmap v627: 168 pgs: 168 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.48k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: 6.c scrub starts 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: 6.c scrub ok 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:33:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:39 smithi181 ceph-mon[33998]: osdmap e285: 8 total, 8 up, 8 in 2024-03-24T11:33:40.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:40.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:40.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:40.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:40.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:40.402 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:40.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:40.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:40.531 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:40.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:40.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 37 16 70223 70207 118.574 67.625 0.00716437 0.00842876 2024-03-24T11:33:40.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:40.674 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:40.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:40.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:40.813 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:40.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:40.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:40.939 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:40.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:41.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:41.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:41.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:40 smithi181 ceph-mon[33998]: 1.10 scrub starts 2024-03-24T11:33:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:40 smithi181 ceph-mon[33998]: 1.10 scrub ok 2024-03-24T11:33:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:40 smithi181 ceph-mon[33998]: 6.e deep-scrub starts 2024-03-24T11:33:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:40 smithi181 ceph-mon[33998]: 6.e deep-scrub ok 2024-03-24T11:33:41.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[29608]: 1.10 scrub starts 2024-03-24T11:33:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[29608]: 1.10 scrub ok 2024-03-24T11:33:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[29608]: 6.e deep-scrub starts 2024-03-24T11:33:41.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[29608]: 6.e deep-scrub ok 2024-03-24T11:33:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[37907]: 1.10 scrub starts 2024-03-24T11:33:41.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[37907]: 1.10 scrub ok 2024-03-24T11:33:41.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[37907]: 6.e deep-scrub starts 2024-03-24T11:33:41.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:40 smithi179 ceph-mon[37907]: 6.e deep-scrub ok 2024-03-24T11:33:41.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:41.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:41.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:41.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:41.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:41.497 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:41.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:41.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:41.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 38 16 71573 71557 117.674 84.375 0.00968986 0.0084929 2024-03-24T11:33:41.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:41.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:41.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:41.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:41.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:41.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:41.913 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:41.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:41.982 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[29608]: pgmap v629: 168 pgs: 168 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:41.982 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[29608]: 4.11 scrub starts 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[29608]: 4.11 scrub ok 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[37907]: pgmap v629: 168 pgs: 168 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[37907]: 4.11 scrub starts 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[37907]: 4.11 scrub ok 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:33:41.983 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:41 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:33:42.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:42.050 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:42.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:41 smithi181 ceph-mon[33998]: pgmap v629: 168 pgs: 168 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:41 smithi181 ceph-mon[33998]: 4.11 scrub starts 2024-03-24T11:33:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:41 smithi181 ceph-mon[33998]: 4.11 scrub ok 2024-03-24T11:33:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:41 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:33:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:41 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:33:42.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:42.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:42.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:42.292 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:42.335 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:42.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:42.437 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:42.497 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:42.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:42.599 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:42.625 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 39 16 72957 72941 116.874 86.5 0.0211316 0.00855216 2024-03-24T11:33:42.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:42.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:42.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:42.798 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:42.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:42.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:42.949 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:42.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:43.012 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:33:43.012 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:33:43.012 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:33:43.012 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:33:43.012 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 3.5 deep-scrub starts 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[29608]: 3.5 deep-scrub ok 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:33:43.013 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 3.5 deep-scrub starts 2024-03-24T11:33:43.014 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:42 smithi179 ceph-mon[37907]: 3.5 deep-scrub ok 2024-03-24T11:33:43.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:43.092 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:43.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 3.5 deep-scrub starts 2024-03-24T11:33:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:42 smithi181 ceph-mon[33998]: 3.5 deep-scrub ok 2024-03-24T11:33:43.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:43.241 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:43.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:43.343 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:43.394 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:43.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:43.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:43.532 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:43.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:43.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:33:43.626343+0000 min lat: 0.00124828 max lat: 0.21968 avg lat: 0.00863335 2024-03-24T11:33:43.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:33:43.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 40 16 74110 74094 115.754 72.0625 0.0131112 0.00863335 2024-03-24T11:33:43.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:43.668 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:43.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:43.759 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:33:43.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:43.809 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:43.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:43.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:43.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:43.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:44.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:44.083 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:44.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: pgmap v630: 168 pgs: 168 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 992 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: 6.b scrub starts 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: 6.b scrub ok 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:33:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/536812053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:33:44.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:33:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:33:44.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: pgmap v630: 168 pgs: 168 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 992 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: 6.b scrub starts 2024-03-24T11:33:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: 6.b scrub ok 2024-03-24T11:33:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/536812053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: pgmap v630: 168 pgs: 168 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 992 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: 6.b scrub starts 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: 6.b scrub ok 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:33:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:33:44.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:33:44.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:33:44.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:43 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/536812053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:33:44.219 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:44.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:44.248 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 10 pg_num to 26 2024-03-24T11:33:44.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:44.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:44.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:44.448 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:33:44.448 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:33:44.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:44.492 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:44.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:44.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:44.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 41 16 75335 75319 114.798 76.5625 0.0132926 0.00870654 2024-03-24T11:33:44.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:44.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:44.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:44.772 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:44.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:44.790 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:44.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:44.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:33:44.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:33:44.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/536812053' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:33:44.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2161310045' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:44.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/536812053' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2161310045' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:44.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:44 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:44.914 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:44.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:45.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:45.049 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:45.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:45.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:33:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:33:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/536812053' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: osdmap e286: 8 total, 8 up, 8 in 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2161310045' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: osdmap e287: 8 total, 8 up, 8 in 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:45.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:44 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:45.189 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:45.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:45.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:33:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:33:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:33:45.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:45.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:45.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:45.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:45.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:33:45.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 42 16 76206 76190 113.361 54.4375 0.110878 0.00876098 2024-03-24T11:33:45.771 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:45.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:45.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:45.898 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:45.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:45 smithi179 ceph-mon[29608]: pgmap v633: 168 pgs: 168 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-24T11:33:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:45 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T11:33:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:45 smithi179 ceph-mon[29608]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T11:33:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:45 smithi179 ceph-mon[37907]: pgmap v633: 168 pgs: 168 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-24T11:33:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:45 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T11:33:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:45 smithi179 ceph-mon[37907]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T11:33:45.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:46.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:46.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.149 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:46.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:45 smithi181 ceph-mon[33998]: pgmap v633: 168 pgs: 168 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-24T11:33:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:45 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T11:33:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:45 smithi181 ceph-mon[33998]: osdmap e288: 8 total, 8 up, 8 in 2024-03-24T11:33:46.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:46.286 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:46.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.428 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:46.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:46.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:46.564 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:46.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 43 16 76744 76728 111.506 33.625 0.00787277 0.00896347 2024-03-24T11:33:46.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.709 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:46.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:46.810 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:46.847 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:46.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:46.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:46.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:47.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:47.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:47.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:47.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:47.267 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:47.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:47.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:47.395 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:47.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[29608]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T11:33:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[29608]: pgmap v636: 178 pgs: 10 unknown, 168 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-24T11:33:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[37907]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T11:33:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[37907]: pgmap v636: 178 pgs: 10 unknown, 168 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-24T11:33:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:47 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:47.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:47.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:47.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:47.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:47.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 44 16 77970 77954 110.714 76.625 0.00302087 0.00896211 2024-03-24T11:33:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:47.646 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:47.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:47.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:47 smithi181 ceph-mon[33998]: osdmap e289: 8 total, 8 up, 8 in 2024-03-24T11:33:47.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:47 smithi181 ceph-mon[33998]: pgmap v636: 178 pgs: 10 unknown, 168 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-24T11:33:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:47 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:47 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T11:33:47.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:47.771 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:47.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:47.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:47.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:47.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:48.000 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:48.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:48.100 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:48.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:48.150 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:48.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:48.277 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:48.369 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:48 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T11:33:48.369 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:48 smithi179 ceph-mon[29608]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T11:33:48.369 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:48 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T11:33:48.369 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:48 smithi179 ceph-mon[37907]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T11:33:48.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:48.402 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:48.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:48.528 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:48.626 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 45 16 77970 77954 108.253 0 - 0.00896211 2024-03-24T11:33:48.630 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:48 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T11:33:48.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:48 smithi181 ceph-mon[33998]: osdmap e290: 8 total, 8 up, 8 in 2024-03-24T11:33:48.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:48.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:48.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:48.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:48.907 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:48.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:49.009 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:49.033 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:49.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:49.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:33:49.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:49.173 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:49.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:49.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:49.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:49.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:49.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:49.443 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:49.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:49.488 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[29608]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T11:33:49.488 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[29608]: pgmap v639: 178 pgs: 1 remapped+peering, 9 unknown, 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 23 MiB/s wr, 370 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-24T11:33:49.488 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:33:49.488 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:49.488 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[29608]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T11:33:49.488 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[37907]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T11:33:49.488 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[37907]: pgmap v639: 178 pgs: 1 remapped+peering, 9 unknown, 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 23 MiB/s wr, 370 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-24T11:33:49.489 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:33:49.489 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:49.489 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:49 smithi179 ceph-mon[37907]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T11:33:49.545 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:49.621 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:49.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:49.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:49.628 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 46 16 77980 77964 105.908 0.3125 2.32018 0.00921002 2024-03-24T11:33:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:49 smithi181 ceph-mon[33998]: osdmap e291: 8 total, 8 up, 8 in 2024-03-24T11:33:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:49 smithi181 ceph-mon[33998]: pgmap v639: 178 pgs: 1 remapped+peering, 9 unknown, 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 23 MiB/s wr, 370 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-24T11:33:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:49 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:33:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:33:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:49 smithi181 ceph-mon[33998]: osdmap e292: 8 total, 8 up, 8 in 2024-03-24T11:33:49.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:49.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:49.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:49.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:49.883 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:49.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:49.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:50.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:50.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:50.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:50.172 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:50.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:50.249 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:33:50.273 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:50.297 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:33:50.332 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:50.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:50.345 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:33:50.357 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:50 smithi179 ceph-mon[29608]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 289'4757 2024-03-24T11:33:50.358 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3512289846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:50.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:50 smithi179 ceph-mon[37907]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 289'4757 2024-03-24T11:33:50.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3512289846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:50.394 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:33:50.419 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.15621.0:78449 10.14 e292/291 10:28d67c4d:::benchmark_data_smithi179_155357_object78448:head v 292'4858, mlcod=0'0)", 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:33:50.324636+0000", 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.016726201, 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016760575999999999, 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.420 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.324636+0000", 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.324636+0000", 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:33:50.421 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.324716+0000", 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9480999999999993e-05 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.333926+0000", 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092102199999999999 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.333944+0000", 2024-03-24T11:33:50.422 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8538000000000001e-05 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.333954+0000", 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.4339999999999998e-06 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.334153+0000", 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019923 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.423 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.334181+0000", 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.8538999999999999e-05 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.15621.0:78454 10.18 e292/291 10:1ef905af:::benchmark_data_smithi179_155357_object78453:head v 292'4926, mlcod=271'22)", 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:33:50.339485+0000", 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0018774830000000001, 2024-03-24T11:33:50.424 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019902140000000001, 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339485+0000", 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339485+0000", 2024-03-24T11:33:50.425 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339486+0000", 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.486e-06 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339522+0000", 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.5638e-05 2024-03-24T11:33:50.426 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339539+0000", 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6997e-05 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339550+0000", 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1442e-05 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:33:50.427 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339751+0000", 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020048700000000001 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339777+0000", 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6528000000000001e-05 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "MOSDPGPush(10.18 292/291 [PushOp(10:1baaa906:::benchmark_data_smithi179_155357_object299:head, version: 271'27, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:1baaa906:::benchmark_data_smithi179_155357_object299:head@271'27, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T11:33:50.428 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:33:50.339582+0000", 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.001780125, 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0019768860000000002, 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339582+0000", 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:50.429 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339582+0000", 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339583+0000", 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.3000000000000001e-07 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.430 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339620+0000", 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6962000000000003e-05 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339641+0000", 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.1036999999999999e-05 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:33:50.431 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:33:50.339657+0000", 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6818999999999999e-05 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 3 2024-03-24T11:33:50.432 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:33:50.435 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:50.440 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:33:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:50.470 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:50.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:50.485 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:33:50.530 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:33:50.569 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:33:50.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:50.577 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:33:50.578 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:33:50.578 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:33:50.578 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:33:50.578 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:33:50.578 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:33:50.578 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:33:50.578 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:33:50.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:50.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:50.629 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 47 16 78713 78697 104.629 45.8125 0.0123647 0.00955126 2024-03-24T11:33:50.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:50 smithi181 ceph-mon[33998]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 289'4757 2024-03-24T11:33:50.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3512289846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:33:50.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:50.746 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:50.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:50.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:50.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:50.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:50.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:51.032 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:51.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:51.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:51.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:51.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:51.245 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:33:51.245 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:33:51.246 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-24T11:33:51.246 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-24T11:33:51.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:33:51.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:51.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:51.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:51.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:51.493 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:51.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:51.594 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:51.629 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 48 16 79618 79602 103.626 56.5625 0.014049 0.00964509 2024-03-24T11:33:51.646 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:51.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:51 smithi181 ceph-mon[33998]: pgmap v641: 178 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 4 remapped+peering, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s; 12336/234504 objects degraded (5.260%); 14485/234504 objects misplaced (6.177%); 1.2 MiB/s, 18 objects/s recovering 2024-03-24T11:33:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:51 smithi179 ceph-mon[29608]: pgmap v641: 178 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 4 remapped+peering, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s; 12336/234504 objects degraded (5.260%); 14485/234504 objects misplaced (6.177%); 1.2 MiB/s, 18 objects/s recovering 2024-03-24T11:33:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:51 smithi179 ceph-mon[37907]: pgmap v641: 178 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 4 remapped+peering, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s; 12336/234504 objects degraded (5.260%); 14485/234504 objects misplaced (6.177%); 1.2 MiB/s, 18 objects/s recovering 2024-03-24T11:33:51.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:51.800 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:51.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:51.902 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:51.934 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:51.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:52.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:52.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:52.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:52.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:52.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:52.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:52.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:52.329 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:52.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:52.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:52.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:52.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:52.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:52.618 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:52.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:52.630 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 49 16 80456 80440 102.58 52.375 0.0389639 0.00974219 2024-03-24T11:33:52.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:52 smithi181 ceph-mon[33998]: Health check failed: Degraded data redundancy: 12336/234504 objects degraded (5.260%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:33:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:52 smithi179 ceph-mon[29608]: Health check failed: Degraded data redundancy: 12336/234504 objects degraded (5.260%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:33:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:52 smithi179 ceph-mon[37907]: Health check failed: Degraded data redundancy: 12336/234504 objects degraded (5.260%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:33:52.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:52.778 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:52.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:52.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:52.925 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:52.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:53.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:53.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:53.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:53.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:53.227 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:53.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:53.328 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:53.366 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:53.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:53.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:53.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:53.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:53.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:53.630 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 50 16 81257 81241 101.53 50.0625 0.0428812 0.00984299 2024-03-24T11:33:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:53 smithi181 ceph-mon[33998]: pgmap v642: 178 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 4 remapped+peering, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 648 op/s; 12336/234504 objects degraded (5.260%); 14485/234504 objects misplaced (6.177%); 821 KiB/s, 12 objects/s recovering 2024-03-24T11:33:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:53 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:53 smithi179 ceph-mon[29608]: pgmap v642: 178 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 4 remapped+peering, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 648 op/s; 12336/234504 objects degraded (5.260%); 14485/234504 objects misplaced (6.177%); 821 KiB/s, 12 objects/s recovering 2024-03-24T11:33:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:53 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:53.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:53 smithi179 ceph-mon[37907]: pgmap v642: 178 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 4 remapped+peering, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 41 MiB/s wr, 648 op/s; 12336/234504 objects degraded (5.260%); 14485/234504 objects misplaced (6.177%); 821 KiB/s, 12 objects/s recovering 2024-03-24T11:33:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:53 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:33:53.736 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:53.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:53.838 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:53.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:53.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:53.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:33:54.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:54.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:54.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:54.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:54.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:54.306 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:54.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:54.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:33:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:33:54.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:54.446 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:54.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:54.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:54.572 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:54.630 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 51 16 81978 81962 100.423 45.0625 0.00830803 0.00995274 2024-03-24T11:33:54.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:54 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:54 smithi181 ceph-mon[33998]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T11:33:54.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:54 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:54 smithi179 ceph-mon[29608]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T11:33:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:54 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:33:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:54 smithi179 ceph-mon[37907]: osdmap e293: 8 total, 8 up, 8 in 2024-03-24T11:33:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.715 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:54.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:54.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:54.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:54.945 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:54.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:54.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:55.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:55.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:33:55.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:55.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:55.213 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:55.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:55.250 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:55.294 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:33:55.304 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:33:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:33:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:33:55.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:55.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:55.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:55.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:55.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:55.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:55.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:55.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:55.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:55 smithi179 ceph-mon[29608]: pgmap v644: 178 pgs: 4 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 22166/241590 objects degraded (9.175%); 23601/241590 objects misplaced (9.769%); 6.3 MiB/s, 101 objects/s recovering 2024-03-24T11:33:55.626 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:55 smithi179 ceph-mon[37907]: pgmap v644: 178 pgs: 4 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 22166/241590 objects degraded (9.175%); 23601/241590 objects misplaced (9.769%); 6.3 MiB/s, 101 objects/s recovering 2024-03-24T11:33:55.630 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 52 16 82687 82671 99.3436 44.3125 0.0142139 0.0100604 2024-03-24T11:33:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:55 smithi181 ceph-mon[33998]: pgmap v644: 178 pgs: 4 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 168 active+clean, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 22166/241590 objects degraded (9.175%); 23601/241590 objects misplaced (9.769%); 6.3 MiB/s, 101 objects/s recovering 2024-03-24T11:33:55.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:55.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:55.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:55.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:55.864 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:55.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:55.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:55.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:55.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:56.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:56.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:56.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:56.232 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:56.283 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:56.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:56.385 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:56.442 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:56.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:56.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:56.577 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:56.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:56.631 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 53 16 83540 83524 98.474 53.3125 0.0391773 0.0101489 2024-03-24T11:33:56.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:56 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 17270/244596 objects degraded (7.061%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:33:56.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:56.694 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:56 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 17270/244596 objects degraded (7.061%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:33:56.694 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:56 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 17270/244596 objects degraded (7.061%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:33:56.708 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:56.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:56.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:56.846 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:56.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:56.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:57.000 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:57.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:57.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:57.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:57.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:57.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:57.287 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:57.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:57.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:57.431 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:57.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:57.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:57.578 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:57.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:57.631 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 54 16 84457 84441 97.7118 57.3125 0.00909828 0.0102273 2024-03-24T11:33:57.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:57 smithi181 ceph-mon[33998]: pgmap v645: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 817 op/s; 17270/244596 objects degraded (7.061%); 27821/244596 objects misplaced (11.374%); 8.7 MiB/s, 139 objects/s recovering 2024-03-24T11:33:57.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:57 smithi179 ceph-mon[29608]: pgmap v645: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 817 op/s; 17270/244596 objects degraded (7.061%); 27821/244596 objects misplaced (11.374%); 8.7 MiB/s, 139 objects/s recovering 2024-03-24T11:33:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:57 smithi179 ceph-mon[37907]: pgmap v645: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 817 op/s; 17270/244596 objects degraded (7.061%); 27821/244596 objects misplaced (11.374%); 8.7 MiB/s, 139 objects/s recovering 2024-03-24T11:33:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:57.726 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:57.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:57.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:33:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:57.883 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:57.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:33:57.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:58.015 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:58.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:58.117 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:33:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:58.158 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:58.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:33:58.259 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:58.305 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:58.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:58.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:58.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:33:58.446 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:58.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:33:58.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:58.606 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:58.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:58.632 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 55 16 85355 85339 96.9536 56.125 0.0320902 0.0103062 2024-03-24T11:33:58.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:33:58.742 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:58.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:33:58.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:58.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:58.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:58.986 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:59.041 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:59.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:59.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:33:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:59.192 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:59.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:33:59.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:59.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:59.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:59.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:59.470 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:59.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:59.485 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:33:59 smithi179 ceph-mon[29608]: pgmap v646: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 565 B/s rd, 98 MiB/s wr, 1.56k op/s; 17270/246885 objects degraded (6.995%); 27257/246885 objects misplaced (11.040%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:33:59.485 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:33:59 smithi179 ceph-mon[37907]: pgmap v646: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 565 B/s rd, 98 MiB/s wr, 1.56k op/s; 17270/246885 objects degraded (6.995%); 27257/246885 objects misplaced (11.040%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:33:59.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:33:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:59.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:59.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:33:59.632 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 56 16 86189 86173 96.1531 52.125 0.0331589 0.0103942 2024-03-24T11:33:59.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:33:59 smithi181 ceph-mon[33998]: pgmap v646: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 565 B/s rd, 98 MiB/s wr, 1.56k op/s; 17270/246885 objects degraded (6.995%); 27257/246885 objects misplaced (11.040%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:33:59.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:33:59.741 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:33:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:59.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:33:59.808 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:33:59.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:33:59.849 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:33:59.885 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:33:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:33:59.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:33:59.898 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:33:59.941 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:33:59.986 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:33:59.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:00.025 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:00.028 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:34:00.078 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:34:00.119 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:34:00.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:00.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:34:00.131 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:34:00.132 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:34:00.132 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:34:00.132 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:34:00.132 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:34:00.132 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:34:00.132 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:34:00.155 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:00.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:00.302 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:00.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.429 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:00.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:00.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:00.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:00.632 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 57 16 86942 86926 95.2919 47.0625 0.00761286 0.0104888 2024-03-24T11:34:00.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/261775245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:34:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:00 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:00 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:00.668 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.707 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:00.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/261775245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:34:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:00 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:00 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/261775245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:34:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:00 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:00 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:00.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:00.845 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:00.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:00.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:00.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:01.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:01.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:01.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:01.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:01.255 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:01.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:01.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:01.389 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:01.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:01.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:01.523 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:01.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:01.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:01.632 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 58 16 87636 87620 94.3968 43.375 0.00334958 0.0105682 2024-03-24T11:34:01.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:01.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:01.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:01 smithi181 ceph-mon[33998]: pgmap v647: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 753 op/s; 17270/257100 objects degraded (6.717%); 26264/257100 objects misplaced (10.215%); 14 MiB/s, 227 objects/s recovering 2024-03-24T11:34:01.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:01 smithi179 ceph-mon[29608]: pgmap v647: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 753 op/s; 17270/257100 objects degraded (6.717%); 26264/257100 objects misplaced (10.215%); 14 MiB/s, 227 objects/s recovering 2024-03-24T11:34:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:01 smithi179 ceph-mon[37907]: pgmap v647: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 753 op/s; 17270/257100 objects degraded (6.717%); 26264/257100 objects misplaced (10.215%); 14 MiB/s, 227 objects/s recovering 2024-03-24T11:34:01.750 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:01.791 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:01.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:01.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:01.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:01.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:02.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:02.068 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:02.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:02.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:02.203 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:02.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:02.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:02.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:02.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:02.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:02.554 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:02.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:02.632 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 59 16 88409 88393 93.6158 48.3125 0.0310722 0.0106761 2024-03-24T11:34:02.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:02.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:02.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:02.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:02.830 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:02.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:02.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:02.961 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:02.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:03.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:03.107 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:03.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:03.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:03.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:03.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:03.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:03.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:03.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:03.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:03.512 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:03.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:03.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:03.633 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:34:03.633409+0000 min lat: 0.00124828 max lat: 2.48431 avg lat: 0.0107691 2024-03-24T11:34:03.633 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:34:03.633 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 60 16 89126 89110 92.8024 44.8125 0.00647717 0.0107691 2024-03-24T11:34:03.642 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:03.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:03.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:03 smithi181 ceph-mon[33998]: pgmap v648: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 753 op/s; 17270/257100 objects degraded (6.717%); 26264/257100 objects misplaced (10.215%); 14 MiB/s, 227 objects/s recovering 2024-03-24T11:34:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:03 smithi179 ceph-mon[29608]: pgmap v648: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 753 op/s; 17270/257100 objects degraded (6.717%); 26264/257100 objects misplaced (10.215%); 14 MiB/s, 227 objects/s recovering 2024-03-24T11:34:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:03 smithi179 ceph-mon[37907]: pgmap v648: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 753 op/s; 17270/257100 objects degraded (6.717%); 26264/257100 objects misplaced (10.215%); 14 MiB/s, 227 objects/s recovering 2024-03-24T11:34:03.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:03.771 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:03.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:03.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:03.907 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:03.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:04.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:04.036 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:04.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:04.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:04.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:04.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:04.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:04.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:34:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:34:04.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:04.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:04.572 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:04.633 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 61 16 89722 89706 91.8918 37.25 0.0145084 0.0108769 2024-03-24T11:34:04.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:04.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:04.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:04.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:04.679 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 17270/257100 objects degraded (6.717%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:04.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:04.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:04.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:04.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:04 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 17270/257100 objects degraded (6.717%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:04.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:04.809 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:04.833 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:04 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:34:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:04 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:04 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 17270/257100 objects degraded (6.717%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:04.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:04.979 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:04.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:05.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:05.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:05.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:05.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:05.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:05.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:05.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:05.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:05.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:05.447 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:34:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:34:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:34:05.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:05.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:05.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:05.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:05.636 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 62 16 90511 90495 91.1997 49.3125 0.016576 0.010958 2024-03-24T11:34:05.637 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:05.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:05 smithi179 ceph-mon[29608]: pgmap v649: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1013 B/s rd, 49 MiB/s wr, 784 op/s; 17270/265344 objects degraded (6.509%); 25133/265344 objects misplaced (9.472%); 17 MiB/s, 270 objects/s recovering 2024-03-24T11:34:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:05 smithi179 ceph-mon[37907]: pgmap v649: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1013 B/s rd, 49 MiB/s wr, 784 op/s; 17270/265344 objects degraded (6.509%); 25133/265344 objects misplaced (9.472%); 17 MiB/s, 270 objects/s recovering 2024-03-24T11:34:05.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:05.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:05.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:05.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:05.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:05 smithi181 ceph-mon[33998]: pgmap v649: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1013 B/s rd, 49 MiB/s wr, 784 op/s; 17270/265344 objects degraded (6.509%); 25133/265344 objects misplaced (9.472%); 17 MiB/s, 270 objects/s recovering 2024-03-24T11:34:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.009 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:06.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:06.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:06.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:06.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:06.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:06.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:06.403 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:06.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:06.533 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:06.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:06.636 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 63 16 90945 90929 90.1828 27.125 0.00533896 0.0110678 2024-03-24T11:34:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:06 smithi179 ceph-mon[29608]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T11:34:06.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:06 smithi179 ceph-mon[29608]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T11:34:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:06 smithi179 ceph-mon[37907]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T11:34:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:06 smithi179 ceph-mon[37907]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T11:34:06.773 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:06.818 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:06 smithi181 ceph-mon[33998]: osdmap e294: 8 total, 8 up, 8 in 2024-03-24T11:34:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:06 smithi181 ceph-mon[33998]: osdmap e295: 8 total, 8 up, 8 in 2024-03-24T11:34:06.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:06.961 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:06.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:07.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:07.106 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:07.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:07.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:07.241 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:07.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:07.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:07.368 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:07.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:07.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:07.520 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:07.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:07.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:07.636 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 64 16 91670 91654 89.4817 45.3125 0.00583894 0.0111694 2024-03-24T11:34:07.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:07.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:07 smithi179 ceph-mon[37907]: pgmap v651: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 788 op/s; 17270/268224 objects degraded (6.439%); 24502/268224 objects misplaced (9.135%); 17 MiB/s, 271 objects/s recovering 2024-03-24T11:34:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:07 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:34:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:07 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:34:07.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:07 smithi179 ceph-mon[29608]: pgmap v651: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 788 op/s; 17270/268224 objects degraded (6.439%); 24502/268224 objects misplaced (9.135%); 17 MiB/s, 271 objects/s recovering 2024-03-24T11:34:07.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:07 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:34:07.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:07 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:34:07.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:07.777 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:07.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:07.879 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:07 smithi181 ceph-mon[33998]: pgmap v651: 178 pgs: 5 active+recovering+undersized+remapped, 168 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 788 op/s; 17270/268224 objects degraded (6.439%); 24502/268224 objects misplaced (9.135%); 17 MiB/s, 271 objects/s recovering 2024-03-24T11:34:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:07 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:34:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:07 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:34:07.926 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:07.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:08.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:08.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:08.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:08.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:08.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:08.317 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:08.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:08.465 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:08.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:08.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:08.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 65 16 92320 92304 88.7302 40.625 0.00518011 0.0112639 2024-03-24T11:34:08.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:08.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:08.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:08.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:08.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:08.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:08.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:09.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:09.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:09.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:09.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:09.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:09.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:09.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:09.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:09.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:09.491 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:09.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:09.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:09.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:09.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 66 16 92946 92930 87.9787 39.125 0.0123676 0.0113603 2024-03-24T11:34:09.678 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:09.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:09.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:09.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:09.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:09.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:09 smithi181 ceph-mon[33998]: pgmap v653: 178 pgs: 4 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 544 op/s; 17270/270150 objects degraded (6.393%); 23964/270150 objects misplaced (8.871%); 16 MiB/s, 252 objects/s recovering 2024-03-24T11:34:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:09 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 17270/270150 objects degraded (6.393%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:09.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:09.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:09 smithi179 ceph-mon[29608]: pgmap v653: 178 pgs: 4 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 544 op/s; 17270/270150 objects degraded (6.393%); 23964/270150 objects misplaced (8.871%); 16 MiB/s, 252 objects/s recovering 2024-03-24T11:34:09.963 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:09 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 17270/270150 objects degraded (6.393%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:09.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:09 smithi179 ceph-mon[37907]: pgmap v653: 178 pgs: 4 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 544 op/s; 17270/270150 objects degraded (6.393%); 23964/270150 objects misplaced (8.871%); 16 MiB/s, 252 objects/s recovering 2024-03-24T11:34:09.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:09 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 17270/270150 objects degraded (6.393%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:09.964 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:09.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:10.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:10.109 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:10.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:10.210 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:10.259 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:10.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:10.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:10.408 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:10.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:10.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:10.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:10.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:10.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:10.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 67 16 93666 93650 87.3372 45 0.0154867 0.0114445 2024-03-24T11:34:10.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:10.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:10.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:10.811 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:10.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:10.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:10.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:10.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:11.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:11.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:11.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:11.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:11.204 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:11.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:11.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:11.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:11.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:11.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:11.501 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:11.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:11.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:11.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:11.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:11.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 68 15 94437 94422 86.7625 48.25 0.00739091 0.0115203 2024-03-24T11:34:11.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:11.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:11.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:11.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:11 smithi181 ceph-mon[33998]: pgmap v654: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 863 op/s; 12278/277800 objects degraded (4.420%); 27612/277800 objects misplaced (9.940%); 22 MiB/s, 356 objects/s recovering 2024-03-24T11:34:11.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:11.909 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:11.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:11 smithi179 ceph-mon[29608]: pgmap v654: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 863 op/s; 12278/277800 objects degraded (4.420%); 27612/277800 objects misplaced (9.940%); 22 MiB/s, 356 objects/s recovering 2024-03-24T11:34:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:11 smithi179 ceph-mon[37907]: pgmap v654: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 863 op/s; 12278/277800 objects degraded (4.420%); 27612/277800 objects misplaced (9.940%); 22 MiB/s, 356 objects/s recovering 2024-03-24T11:34:12.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:12.033 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:12.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:12.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:12.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:12.304 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:12.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:12.435 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:12.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:12.580 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:12.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 69 16 95185 95169 86.1817 46.6875 0.0222864 0.0115977 2024-03-24T11:34:12.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:12.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:12.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:12.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:12.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:12.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:12.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:13.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:13.128 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:13.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:13.230 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:13.261 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:13.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:13.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:13.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:13.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:13.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:13.527 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:13.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:13.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:13.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 70 16 95923 95907 85.6095 46.125 0.0169833 0.0116749 2024-03-24T11:34:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:13.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:13.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:13.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:13.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:13.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:13.901 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:34:13.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:13 smithi179 ceph-mon[29608]: pgmap v655: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 519 op/s; 12278/277800 objects degraded (4.420%); 27612/277800 objects misplaced (9.940%); 14 MiB/s, 231 objects/s recovering 2024-03-24T11:34:13.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:13 smithi179 ceph-mon[37907]: pgmap v655: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 519 op/s; 12278/277800 objects degraded (4.420%); 27612/277800 objects misplaced (9.940%); 14 MiB/s, 231 objects/s recovering 2024-03-24T11:34:13.905 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:13 smithi181 ceph-mon[33998]: pgmap v655: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 519 op/s; 12278/277800 objects degraded (4.420%); 27612/277800 objects misplaced (9.940%); 14 MiB/s, 231 objects/s recovering 2024-03-24T11:34:13.915 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:34:13.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:13.969 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:13.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:14.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:14.104 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:14.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:14.205 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:14.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:14.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:14.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:14.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:14.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:14.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:34:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:34:14.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:14.505 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:14.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:14.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:14.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:14.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:14.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 71 16 96746 96730 85.1282 51.4375 0.0332401 0.0117414 2024-03-24T11:34:14.738 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:14.789 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:14.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:14.827 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:14 smithi179 ceph-mon[37907]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:34:14.827 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:14 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:14.827 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:14 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 12278/277800 objects degraded (4.420%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:14.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:14 smithi179 ceph-mon[29608]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:34:14.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:14 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:14.828 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:14 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 12278/277800 objects degraded (4.420%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:14.891 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:14 smithi181 ceph-mon[33998]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:34:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:14 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:14 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 12278/277800 objects degraded (4.420%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:14.931 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:14.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:15.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:15.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:15.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:15.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:15.204 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:15.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:15.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:34:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:34:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:34:15.305 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:15.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:15.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:15.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:15.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:15.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:15.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:15.617 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:15.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:15.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 72 16 97405 97389 84.518 41.1875 0.00643598 0.0118216 2024-03-24T11:34:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:15 smithi179 ceph-mon[37907]: pgmap v656: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 666 op/s; 12278/285993 objects degraded (4.293%); 27334/285993 objects misplaced (9.558%); 11 MiB/s, 169 objects/s recovering 2024-03-24T11:34:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:15 smithi179 ceph-mon[29608]: pgmap v656: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 666 op/s; 12278/285993 objects degraded (4.293%); 27334/285993 objects misplaced (9.558%); 11 MiB/s, 169 objects/s recovering 2024-03-24T11:34:15.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:15.761 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:15.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:15.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:15.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:15.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:15.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:15.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:15 smithi181 ceph-mon[33998]: pgmap v656: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 666 op/s; 12278/285993 objects degraded (4.293%); 27334/285993 objects misplaced (9.558%); 11 MiB/s, 169 objects/s recovering 2024-03-24T11:34:15.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:16.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:16.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:16.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:16.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:16.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:16.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:16.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:16.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:16.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:16.428 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:16.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:16.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:16.574 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:16.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:16.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 73 16 98213 98197 84.052 50.5 0.0145652 0.0118925 2024-03-24T11:34:16.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:16.721 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:16.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:16.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:16.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:16.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:16.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:17.010 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:17.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:17.141 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:17.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:17.281 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:17.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:17.413 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:17.515 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.557 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:17.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:17.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 74 16 98972 98956 83.5572 47.4375 0.00733213 0.0119505 2024-03-24T11:34:17.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.718 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:17.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:17.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.854 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:17.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:17 smithi181 ceph-mon[33998]: pgmap v657: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 43 MiB/s wr, 685 op/s; 12278/288771 objects degraded (4.252%); 26304/288771 objects misplaced (9.109%); 14 MiB/s, 217 objects/s recovering 2024-03-24T11:34:17.954 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:17 smithi179 ceph-mon[37907]: pgmap v657: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 43 MiB/s wr, 685 op/s; 12278/288771 objects degraded (4.252%); 26304/288771 objects misplaced (9.109%); 14 MiB/s, 217 objects/s recovering 2024-03-24T11:34:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:17 smithi179 ceph-mon[29608]: pgmap v657: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 43 MiB/s wr, 685 op/s; 12278/288771 objects degraded (4.252%); 26304/288771 objects misplaced (9.109%); 14 MiB/s, 217 objects/s recovering 2024-03-24T11:34:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:18.008 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:18.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:18.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:18.182 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:18.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:18.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:18.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:18.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:18.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:18.485 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:18.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:18.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:18.613 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:18.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:18.637 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 75 16 99760 99744 83.0998 49.25 0.00442499 0.0120216 2024-03-24T11:34:18.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:18.755 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:18.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:18.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:18.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:18 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:18.903 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:18.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:18.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:18 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:18 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:19.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:19.032 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:19.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:19.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:19.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:19.310 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:19.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:19.454 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:19.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:19.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:19.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 76 16 100544 100528 82.6511 49 0.0162855 0.012094 2024-03-24T11:34:19.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:19.717 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:19.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:19.859 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:19.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:19 smithi181 ceph-mon[33998]: pgmap v658: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 860 B/s rd, 40 MiB/s wr, 636 op/s; 12278/290925 objects degraded (4.220%); 26304/290925 objects misplaced (9.042%); 11 MiB/s, 182 objects/s recovering 2024-03-24T11:34:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:19 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 12278/290925 objects degraded (4.220%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:19 smithi179 ceph-mon[37907]: pgmap v658: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 860 B/s rd, 40 MiB/s wr, 636 op/s; 12278/290925 objects degraded (4.220%); 26304/290925 objects misplaced (9.042%); 11 MiB/s, 182 objects/s recovering 2024-03-24T11:34:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:19 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 12278/290925 objects degraded (4.220%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:19 smithi179 ceph-mon[29608]: pgmap v658: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 860 B/s rd, 40 MiB/s wr, 636 op/s; 12278/290925 objects degraded (4.220%); 26304/290925 objects misplaced (9.042%); 11 MiB/s, 182 objects/s recovering 2024-03-24T11:34:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:19 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 12278/290925 objects degraded (4.220%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:19.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:19.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:19.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:20.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:20.124 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:20.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:20.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:20.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:20.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:20.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:20.395 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:20.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:20.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:20.551 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:20.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:20.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 77 16 101097 101081 82.0267 34.5625 0.0138526 0.0121863 2024-03-24T11:34:20.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:20.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:20.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:20.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:20.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:20.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:20.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:20.959 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:20.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:21.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:21.091 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:21.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:21.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:21.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:21.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:21.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:21.345 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:21.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:21.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:21.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:21.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:21.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:21.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:21.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:21.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 78 16 101757 101741 81.5039 41.25 0.00830175 0.0122644 2024-03-24T11:34:21.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:21.747 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:21.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:21.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:21.874 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:21.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:21 smithi181 ceph-mon[33998]: pgmap v659: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s; 12278/300396 objects degraded (4.087%); 24952/300396 objects misplaced (8.306%); 13 MiB/s, 207 objects/s recovering 2024-03-24T11:34:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:21 smithi179 ceph-mon[37907]: pgmap v659: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s; 12278/300396 objects degraded (4.087%); 24952/300396 objects misplaced (8.306%); 13 MiB/s, 207 objects/s recovering 2024-03-24T11:34:21.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:21 smithi179 ceph-mon[29608]: pgmap v659: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s; 12278/300396 objects degraded (4.087%); 24952/300396 objects misplaced (8.306%); 13 MiB/s, 207 objects/s recovering 2024-03-24T11:34:21.976 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:22.031 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:22.132 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:22.178 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:22.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:22.307 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:22.409 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:22.439 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:22.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:22.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:22.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 79 16 102597 102581 81.1367 52.5 0.00872255 0.0123173 2024-03-24T11:34:22.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:22.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:22.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:22.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:22.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:22.969 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:22.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:23.071 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:23.121 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:23.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:23.222 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:23.269 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:23.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:23.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:23.406 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:23.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:23.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:23.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:23.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:23.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:23.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:34:23.638843+0000 min lat: 0.00124828 max lat: 2.48431 avg lat: 0.0123903 2024-03-24T11:34:23.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:34:23.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 80 16 103266 103250 80.6452 41.8125 0.00671791 0.0123903 2024-03-24T11:34:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:23.670 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:23.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:23.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:23.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:23.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:23.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:23 smithi181 ceph-mon[33998]: pgmap v660: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 39 MiB/s wr, 628 op/s; 12278/300396 objects degraded (4.087%); 24952/300396 objects misplaced (8.306%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T11:34:23.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:23.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:23.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:23 smithi179 ceph-mon[29608]: pgmap v660: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 39 MiB/s wr, 628 op/s; 12278/300396 objects degraded (4.087%); 24952/300396 objects misplaced (8.306%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T11:34:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:23 smithi179 ceph-mon[37907]: pgmap v660: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 39 MiB/s wr, 628 op/s; 12278/300396 objects degraded (4.087%); 24952/300396 objects misplaced (8.306%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T11:34:24.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:24.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:24.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:24.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:24.331 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.355 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:24.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:24.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:34:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:34:24.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:24.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:24.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:24.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:24.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 81 16 103986 103970 80.2052 45 0.0933217 0.0124529 2024-03-24T11:34:24.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:24.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:24.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:24.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:24.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:24.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:24 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:24 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 12278/300396 objects degraded (4.087%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:24 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:24 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 12278/300396 objects degraded (4.087%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:24.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:24 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:24.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:24 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 12278/300396 objects degraded (4.087%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:24.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:25.042 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:25.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.180 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:25.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:25.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:25.315 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:25.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:25.449 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:34:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:34:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:34:25.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:25.584 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:25.638 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 82 16 104746 104730 79.8063 47.5 0.0122869 0.0125251 2024-03-24T11:34:25.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.733 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:25.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:25.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:25.857 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:25.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:25 smithi181 ceph-mon[33998]: pgmap v661: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 852 op/s; 12278/308451 objects degraded (3.981%); 24670/308451 objects misplaced (7.998%); 9.4 MiB/s, 149 objects/s recovering 2024-03-24T11:34:25.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:25.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:25 smithi179 ceph-mon[37907]: pgmap v661: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 852 op/s; 12278/308451 objects degraded (3.981%); 24670/308451 objects misplaced (7.998%); 9.4 MiB/s, 149 objects/s recovering 2024-03-24T11:34:25.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:25 smithi179 ceph-mon[29608]: pgmap v661: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 852 op/s; 12278/308451 objects degraded (3.981%); 24670/308451 objects misplaced (7.998%); 9.4 MiB/s, 149 objects/s recovering 2024-03-24T11:34:25.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:25.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:26.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:26.131 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:26.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:26.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:26.261 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:26.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:26.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:26.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:26.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:26.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:26.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:26.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:26.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:26.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 83 16 105450 105434 79.3742 44 0.0260405 0.012593 2024-03-24T11:34:26.663 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:26.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:26.765 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:26.809 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:26.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:26.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:26.951 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:26.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:26.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:27.053 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:27.084 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:27.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:27.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:27.237 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:27.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:27.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:27.394 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:27.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:27.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:27.536 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:27.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:27.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 84 16 106191 106175 78.9806 46.3125 0.008911 0.012656 2024-03-24T11:34:27.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:27.685 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:27.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[29608]: pgmap v662: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 44 MiB/s wr, 697 op/s; 12278/311094 objects degraded (3.947%); 23430/311094 objects misplaced (7.531%); 13 MiB/s, 208 objects/s recovering 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[37907]: pgmap v662: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 44 MiB/s wr, 697 op/s; 12278/311094 objects degraded (3.947%); 23430/311094 objects misplaced (7.531%); 13 MiB/s, 208 objects/s recovering 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:34:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:34:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:34:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:27 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:34:27.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:27.832 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:27.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:27 smithi181 ceph-mon[33998]: pgmap v662: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 767 B/s rd, 44 MiB/s wr, 697 op/s; 12278/311094 objects degraded (3.947%); 23430/311094 objects misplaced (7.531%); 13 MiB/s, 208 objects/s recovering 2024-03-24T11:34:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:27 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:34:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:27 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:34:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:27 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:34:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:27 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:34:27.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:27.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:28.067 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:28.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:28.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:28.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:28.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:28.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:28.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:28.394 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:28.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:28.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:28.519 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:28.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:28.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:28.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 85 16 106973 106957 78.6264 48.875 0.00876453 0.0127131 2024-03-24T11:34:28.662 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:28.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:28.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:28.809 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:28.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:28 smithi181 ceph-mon[33998]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T11:34:28.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:28.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:28.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:28.955 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:28 smithi179 ceph-mon[29608]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T11:34:28.955 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:28 smithi179 ceph-mon[37907]: osdmap e296: 8 total, 8 up, 8 in 2024-03-24T11:34:29.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:29.073 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:29.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:29.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:29.209 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:29.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:29.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:29.355 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:29.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:29.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:29.496 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:29.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:29.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:29.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:29.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:29.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 86 16 107748 107732 78.2754 48.4375 0.00461391 0.0127541 2024-03-24T11:34:29.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:29.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:29.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:29.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:29.897 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:29.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:29.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:29 smithi181 ceph-mon[33998]: pgmap v664: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 921 B/s rd, 46 MiB/s wr, 743 op/s; 12278/313215 objects degraded (3.920%); 23430/313215 objects misplaced (7.480%); 11 MiB/s, 183 objects/s recovering 2024-03-24T11:34:29.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:29 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 12278/313215 objects degraded (3.920%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:29 smithi181 ceph-mon[33998]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T11:34:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:29 smithi179 ceph-mon[29608]: pgmap v664: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 921 B/s rd, 46 MiB/s wr, 743 op/s; 12278/313215 objects degraded (3.920%); 23430/313215 objects misplaced (7.480%); 11 MiB/s, 183 objects/s recovering 2024-03-24T11:34:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:29 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 12278/313215 objects degraded (3.920%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:29 smithi179 ceph-mon[29608]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T11:34:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:29 smithi179 ceph-mon[37907]: pgmap v664: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 921 B/s rd, 46 MiB/s wr, 743 op/s; 12278/313215 objects degraded (3.920%); 23430/313215 objects misplaced (7.480%); 11 MiB/s, 183 objects/s recovering 2024-03-24T11:34:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:29 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 12278/313215 objects degraded (3.920%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:29 smithi179 ceph-mon[37907]: osdmap e297: 8 total, 8 up, 8 in 2024-03-24T11:34:29.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:30.030 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:30.132 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.165 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:30.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:30.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:30.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:30.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:30.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:30.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:30.552 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:30.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 87 16 108149 108133 77.6638 25.0625 0.0110712 0.0128709 2024-03-24T11:34:30.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:30.693 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:30.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:30.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:30.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:30.955 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:30.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:31.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:34:31.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:31.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:31.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:31.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:31.209 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:31.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:31.310 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:31.358 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:31.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:31.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:31 smithi179 ceph-mon[29608]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T11:34:31.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:31 smithi179 ceph-mon[29608]: pgmap v667: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 741 op/s; 12278/321795 objects degraded (3.815%); 22108/321795 objects misplaced (6.870%); 16 MiB/s, 249 objects/s recovering 2024-03-24T11:34:31.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:31 smithi179 ceph-mon[37907]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T11:34:31.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:31 smithi179 ceph-mon[37907]: pgmap v667: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 741 op/s; 12278/321795 objects degraded (3.815%); 22108/321795 objects misplaced (6.870%); 16 MiB/s, 249 objects/s recovering 2024-03-24T11:34:31.459 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:31.499 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:31.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:31.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:31.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:31.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:31.639 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 88 16 108832 108816 77.2663 42.6875 0.0360189 0.0129373 2024-03-24T11:34:31.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:31 smithi181 ceph-mon[33998]: osdmap e298: 8 total, 8 up, 8 in 2024-03-24T11:34:31.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:31 smithi181 ceph-mon[33998]: pgmap v667: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 741 op/s; 12278/321795 objects degraded (3.815%); 22108/321795 objects misplaced (6.870%); 16 MiB/s, 249 objects/s recovering 2024-03-24T11:34:31.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:31.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:31.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:31.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:31.922 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:31.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:32.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:32.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:32.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:32.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:32.202 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:32.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:32.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:32.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:32.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:32.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:32.496 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:32.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:32.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:32.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:32.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:32.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 89 16 109612 109596 76.9459 48.75 0.00490597 0.0129784 2024-03-24T11:34:32.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:32.766 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:32.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:32.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:32.910 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:32.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:33.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:33.038 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:33.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:33.140 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:33.180 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:33.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:33.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:33.326 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:33.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:33.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:33.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:33.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:33.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:33.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:33.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total time run: 90.0124 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total writes made: 110493 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Write size: 65536 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object size: 65536 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Bandwidth (MB/sec): 76.7206 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Bandwidth: 43.625 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max bandwidth (MB/sec): 212.75 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min bandwidth (MB/sec): 0 2024-03-24T11:34:33.640 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average IOPS: 1227 2024-03-24T11:34:33.641 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev IOPS: 697.999 2024-03-24T11:34:33.641 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max IOPS: 3404 2024-03-24T11:34:33.641 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min IOPS: 0 2024-03-24T11:34:33.641 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average Latency(s): 0.0130298 2024-03-24T11:34:33.641 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Latency(s): 0.0348772 2024-03-24T11:34:33.641 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max latency(s): 2.48431 2024-03-24T11:34:33.641 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min latency(s): 0.00124828 2024-03-24T11:34:33.645 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T11:34:33.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:33 smithi181 ceph-mon[33998]: pgmap v668: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 37 MiB/s wr, 595 op/s; 12278/321795 objects degraded (3.815%); 22108/321795 objects misplaced (6.870%); 6.8 MiB/s, 109 objects/s recovering 2024-03-24T11:34:33.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:33 smithi179 ceph-mon[29608]: pgmap v668: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 37 MiB/s wr, 595 op/s; 12278/321795 objects degraded (3.815%); 22108/321795 objects misplaced (6.870%); 6.8 MiB/s, 109 objects/s recovering 2024-03-24T11:34:33.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:33 smithi179 ceph-mon[37907]: pgmap v668: 178 pgs: 5 active+recovering+undersized+remapped, 169 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 37 MiB/s wr, 595 op/s; 12278/321795 objects degraded (3.815%); 22108/321795 objects misplaced (6.870%); 6.8 MiB/s, 109 objects/s recovering 2024-03-24T11:34:33.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:33.742 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:33.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:33.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:33.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:33.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:34.029 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:34.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:34.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:34.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:34.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.305 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:34.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:34.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:34:33 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:34:34.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:34.456 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:34.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:34.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:34.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 12278/321795 objects degraded (3.815%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[29608]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T11:34:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 12278/321795 objects degraded (3.815%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:34 smithi179 ceph-mon[37907]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T11:34:34.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:34.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:34.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:34 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:34 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:34 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 12278/321795 objects degraded (3.815%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:34.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:34 smithi181 ceph-mon[33998]: osdmap e299: 8 total, 8 up, 8 in 2024-03-24T11:34:34.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:34.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:34.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:35.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:35.143 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:35.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:35.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:35.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:35.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:35.376 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:35.409 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:34:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:34:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:34:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:35.420 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:35.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:35.521 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:35.572 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:35.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:35.673 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:35 smithi179 ceph-mon[29608]: pgmap v670: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 61 MiB/s wr, 975 op/s; 12278/330762 objects degraded (3.712%); 21080/330762 objects misplaced (6.373%); 14 MiB/s, 228 objects/s recovering 2024-03-24T11:34:35.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:35 smithi179 ceph-mon[37907]: pgmap v670: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 61 MiB/s wr, 975 op/s; 12278/330762 objects degraded (3.712%); 21080/330762 objects misplaced (6.373%); 14 MiB/s, 228 objects/s recovering 2024-03-24T11:34:35.721 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:35.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:35.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:35.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:35.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:35.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:35.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:35 smithi181 ceph-mon[33998]: pgmap v670: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.5 KiB/s rd, 61 MiB/s wr, 975 op/s; 12278/330762 objects degraded (3.712%); 21080/330762 objects misplaced (6.373%); 14 MiB/s, 228 objects/s recovering 2024-03-24T11:34:35.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:35.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:35.999 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:35.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:36.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:36.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:36.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:36.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:36.258 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:36.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:36.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:36.408 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:36.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:36.438 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:34:36.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:36.551 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:36.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:36.652 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:36.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:34:36.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:34:36.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[29608]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T11:34:36.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3332951211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:34:36.670 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:34:36.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:34:36.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:34:36.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[37907]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T11:34:36.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3332951211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:34:36.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:36 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:34:36.691 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:36.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:36.727 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2024-03-24T11:34:12.331952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6056253790,"num_objects":92600,"num_object_clones":0,"num_object_copies":277800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12278,"num_objects_misplaced":27612,"num_objects_unfound":0,"num_objects_dirty":92600,"num_whiteouts":0,"num_read":1370,"num_read_kb":1122,"num_write":93124,"num_write_kb":5913773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5233,"num_bytes_recovered":341115430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134998,"ondisk_log_size":134998,"up":534,"acting":526,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":553,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22079216,"kb_used_data":18070300,"kb_used_omap":12,"kb_used_meta":4008819,"kb_avail":727882000,"statfs":{"total":767960285184,"available":745351168000,"internally_reserved":0,"allocated":18503987200,"data_stored":18500398042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4105031217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":272105472,"num_objects":4152,"num_object_clones":0,"num_object_copies":12456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4992,"num_objects_misplaced":2479,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4152,"num_write_kb":265728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":121634816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003799"},"pg_stats":[{"pgid":"10.2","version":"295'5250","reported_seq":5324,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:09.058986+0000","last_change":"2024-03-24T11:33:46.314702+0000","last_active":"2024-03-24T11:34:09.058986+0000","last_peered":"2024-03-24T11:34:09.058986+0000","last_clean":"2024-03-24T11:34:09.058986+0000","last_became_active":"2024-03-24T11:33:46.312428+0000","last_became_peered":"2024-03-24T11:33:46.312428+0000","last_unstale":"2024-03-24T11:34:09.058986+0000","last_undegraded":"2024-03-24T11:34:09.058986+0000","last_fullsized":"2024-03-24T11:34:09.058986+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5250,"log_dups_size":0,"ondisk_log_size":5250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:09.240596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188645376,"num_objects":2878,"num_object_clones":0,"num_object_copies":8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2878,"num_write_kb":184224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"213'20","reported_seq":549,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441562+0000","last_change":"2024-03-24T11:32:36.799202+0000","last_active":"2024-03-24T11:34:06.441562+0000","last_peered":"2024-03-24T11:34:06.441562+0000","last_clean":"2024-03-24T11:34:06.441562+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:34:06.441562+0000","last_undegraded":"2024-03-24T11:34:06.441562+0000","last_fullsized":"2024-03-24T11:34:06.441562+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:36.799084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.799084+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":515,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.455919+0000","last_change":"2024-03-24T11:33:25.353540+0000","last_active":"2024-03-24T11:34:06.455919+0000","last_peered":"2024-03-24T11:34:06.455919+0000","last_clean":"2024-03-24T11:34:06.455919+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:34:06.455919+0000","last_undegraded":"2024-03-24T11:34:06.455919+0000","last_fullsized":"2024-03-24T11:34:06.455919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:25.353054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:25.353054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:31.524907+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":730,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.443974+0000","last_change":"2024-03-24T11:32:37.091705+0000","last_active":"2024-03-24T11:34:06.443974+0000","last_peered":"2024-03-24T11:34:06.443974+0000","last_clean":"2024-03-24T11:34:06.443974+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:34:06.443974+0000","last_undegraded":"2024-03-24T11:34:06.443974+0000","last_fullsized":"2024-03-24T11:34:06.443974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:37.091630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.091630+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442343+0000","last_change":"2024-03-24T11:33:39.983218+0000","last_active":"2024-03-24T11:34:06.442343+0000","last_peered":"2024-03-24T11:34:06.442343+0000","last_clean":"2024-03-24T11:34:06.442343+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:34:06.442343+0000","last_undegraded":"2024-03-24T11:34:06.442343+0000","last_fullsized":"2024-03-24T11:34:06.442343+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_clean_scrub_stamp":"2024-03-24T11:33:39.980621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:09.761165+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":569,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.455160+0000","last_change":"2024-03-24T11:31:10.289542+0000","last_active":"2024-03-24T11:34:06.455160+0000","last_peered":"2024-03-24T11:34:06.455160+0000","last_clean":"2024-03-24T11:34:06.455160+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:34:06.455160+0000","last_undegraded":"2024-03-24T11:34:06.455160+0000","last_fullsized":"2024-03-24T11:34:06.455160+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:10.289309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.289309+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"295'5397","reported_seq":5475,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.842000+0000","last_change":"2024-03-24T11:33:46.331280+0000","last_active":"2024-03-24T11:34:08.842000+0000","last_peered":"2024-03-24T11:34:08.842000+0000","last_clean":"2024-03-24T11:34:08.842000+0000","last_became_active":"2024-03-24T11:33:46.302266+0000","last_became_peered":"2024-03-24T11:33:46.302266+0000","last_unstale":"2024-03-24T11:34:08.842000+0000","last_undegraded":"2024-03-24T11:34:08.842000+0000","last_fullsized":"2024-03-24T11:34:08.842000+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:10.011658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194904064,"num_objects":2974,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2974,"num_write_kb":190336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"213'39","reported_seq":608,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442198+0000","last_change":"2024-03-24T11:32:42.683845+0000","last_active":"2024-03-24T11:34:06.442198+0000","last_peered":"2024-03-24T11:34:06.442198+0000","last_clean":"2024-03-24T11:34:06.442198+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:34:06.442198+0000","last_undegraded":"2024-03-24T11:34:06.442198+0000","last_fullsized":"2024-03-24T11:34:06.442198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:32:42.683738+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.683738+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":555,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445375+0000","last_change":"2024-03-24T11:32:23.906538+0000","last_active":"2024-03-24T11:34:06.445375+0000","last_peered":"2024-03-24T11:34:06.445375+0000","last_clean":"2024-03-24T11:34:06.445375+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:34:06.445375+0000","last_undegraded":"2024-03-24T11:34:06.445375+0000","last_fullsized":"2024-03-24T11:34:06.445375+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:32:23.906455+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.906455+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437608+0000","last_change":"2024-03-24T11:32:25.619271+0000","last_active":"2024-03-24T11:34:06.437608+0000","last_peered":"2024-03-24T11:34:06.437608+0000","last_clean":"2024-03-24T11:34:06.437608+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:34:06.437608+0000","last_undegraded":"2024-03-24T11:34:06.437608+0000","last_fullsized":"2024-03-24T11:34:06.437608+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:25.619184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.619184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":535,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445385+0000","last_change":"2024-03-24T11:33:26.128776+0000","last_active":"2024-03-24T11:34:06.445385+0000","last_peered":"2024-03-24T11:34:06.445385+0000","last_clean":"2024-03-24T11:34:06.445385+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:34:06.445385+0000","last_undegraded":"2024-03-24T11:34:06.445385+0000","last_fullsized":"2024-03-24T11:34:06.445385+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:26.128656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:26.128656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:55.453559+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":666,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437400+0000","last_change":"2024-03-24T11:31:31.621753+0000","last_active":"2024-03-24T11:34:06.437400+0000","last_peered":"2024-03-24T11:34:06.437400+0000","last_clean":"2024-03-24T11:34:06.437400+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:34:06.437400+0000","last_undegraded":"2024-03-24T11:34:06.437400+0000","last_fullsized":"2024-03-24T11:34:06.437400+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.621646+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"295'5139","reported_seq":5213,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.292741+0000","last_change":"2024-03-24T11:33:46.287097+0000","last_active":"2024-03-24T11:34:08.292741+0000","last_peered":"2024-03-24T11:34:08.292741+0000","last_clean":"2024-03-24T11:34:08.292741+0000","last_became_active":"2024-03-24T11:33:46.284906+0000","last_became_peered":"2024-03-24T11:33:46.284906+0000","last_unstale":"2024-03-24T11:34:08.292741+0000","last_undegraded":"2024-03-24T11:34:08.292741+0000","last_fullsized":"2024-03-24T11:34:08.292741+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:12.885501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":2816,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2816,"num_write_kb":180224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"213'19","reported_seq":558,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441540+0000","last_change":"2024-03-24T11:32:37.784109+0000","last_active":"2024-03-24T11:34:06.441540+0000","last_peered":"2024-03-24T11:34:06.441540+0000","last_clean":"2024-03-24T11:34:06.441540+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:34:06.441540+0000","last_undegraded":"2024-03-24T11:34:06.441540+0000","last_fullsized":"2024-03-24T11:34:06.441540+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:37.784011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.784011+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":524,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442126+0000","last_change":"2024-03-24T11:31:29.730215+0000","last_active":"2024-03-24T11:34:06.442126+0000","last_peered":"2024-03-24T11:34:06.442126+0000","last_clean":"2024-03-24T11:34:06.442126+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:34:06.442126+0000","last_undegraded":"2024-03-24T11:34:06.442126+0000","last_fullsized":"2024-03-24T11:34:06.442126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.730078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.730078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":523,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438565+0000","last_change":"2024-03-24T11:33:18.355226+0000","last_active":"2024-03-24T11:34:06.438565+0000","last_peered":"2024-03-24T11:34:06.438565+0000","last_clean":"2024-03-24T11:34:06.438565+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:34:06.438565+0000","last_undegraded":"2024-03-24T11:34:06.438565+0000","last_fullsized":"2024-03-24T11:34:06.438565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:18.355107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:33:18.355107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:31.561790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":503,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.458688+0000","last_change":"2024-03-24T11:33:38.472649+0000","last_active":"2024-03-24T11:34:06.458688+0000","last_peered":"2024-03-24T11:34:06.458688+0000","last_clean":"2024-03-24T11:34:06.458688+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:34:06.458688+0000","last_undegraded":"2024-03-24T11:34:06.458688+0000","last_fullsized":"2024-03-24T11:34:06.458688+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:38.472418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.472418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:51.655174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":685,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.439141+0000","last_change":"2024-03-24T11:32:24.651691+0000","last_active":"2024-03-24T11:34:06.439141+0000","last_peered":"2024-03-24T11:34:06.439141+0000","last_clean":"2024-03-24T11:34:06.439141+0000","last_became_active":"2024-03-24T11:28:43.190009+0000","last_became_peered":"2024-03-24T11:28:43.190009+0000","last_unstale":"2024-03-24T11:34:06.439141+0000","last_undegraded":"2024-03-24T11:34:06.439141+0000","last_fullsized":"2024-03-24T11:34:06.439141+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:24.651587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.651587+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"295'5208","reported_seq":5282,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.269410+0000","last_change":"2024-03-24T11:33:46.279776+0000","last_active":"2024-03-24T11:34:08.269410+0000","last_peered":"2024-03-24T11:34:08.269410+0000","last_clean":"2024-03-24T11:34:08.269410+0000","last_became_active":"2024-03-24T11:33:46.277468+0000","last_became_peered":"2024-03-24T11:33:46.277468+0000","last_unstale":"2024-03-24T11:34:08.269410+0000","last_undegraded":"2024-03-24T11:34:08.269410+0000","last_fullsized":"2024-03-24T11:34:08.269410+0000","mapping_epoch":267,"log_start":"272'600","ondisk_log_start":"272'600","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:09.939966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187170816,"num_objects":2856,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":182784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"213'34","reported_seq":585,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.439248+0000","last_change":"2024-03-24T11:32:30.766638+0000","last_active":"2024-03-24T11:34:06.439248+0000","last_peered":"2024-03-24T11:34:06.439248+0000","last_clean":"2024-03-24T11:34:06.439248+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:34:06.439248+0000","last_undegraded":"2024-03-24T11:34:06.439248+0000","last_fullsized":"2024-03-24T11:34:06.439248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:32:30.766549+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.766549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":516,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.454758+0000","last_change":"2024-03-24T11:31:22.167292+0000","last_active":"2024-03-24T11:34:06.454758+0000","last_peered":"2024-03-24T11:34:06.454758+0000","last_clean":"2024-03-24T11:34:06.454758+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:34:06.454758+0000","last_undegraded":"2024-03-24T11:34:06.454758+0000","last_fullsized":"2024-03-24T11:34:06.454758+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.167171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.167171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":706,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442244+0000","last_change":"2024-03-24T11:32:18.783487+0000","last_active":"2024-03-24T11:34:06.442244+0000","last_peered":"2024-03-24T11:34:06.442244+0000","last_clean":"2024-03-24T11:34:06.442244+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:34:06.442244+0000","last_undegraded":"2024-03-24T11:34:06.442244+0000","last_fullsized":"2024-03-24T11:34:06.442244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:18.783339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.783339+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":511,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437996+0000","last_change":"2024-03-24T11:31:28.690070+0000","last_active":"2024-03-24T11:34:06.437996+0000","last_peered":"2024-03-24T11:34:06.437996+0000","last_clean":"2024-03-24T11:34:06.437996+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:34:06.437996+0000","last_undegraded":"2024-03-24T11:34:06.437996+0000","last_fullsized":"2024-03-24T11:34:06.437996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.689946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.689946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":665,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437622+0000","last_change":"2024-03-24T11:33:21.341057+0000","last_active":"2024-03-24T11:34:06.437622+0000","last_peered":"2024-03-24T11:34:06.437622+0000","last_clean":"2024-03-24T11:34:06.437622+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:34:06.437622+0000","last_undegraded":"2024-03-24T11:34:06.437622+0000","last_fullsized":"2024-03-24T11:34:06.437622+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:21.340939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:21.340939+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:25.744920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"295'5779","reported_seq":5848,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.269270+0000","last_change":"2024-03-24T11:32:52.325471+0000","last_active":"2024-03-24T11:34:08.269270+0000","last_peered":"2024-03-24T11:34:08.269270+0000","last_clean":"2024-03-24T11:34:08.269270+0000","last_became_active":"2024-03-24T11:32:52.325161+0000","last_became_peered":"2024-03-24T11:32:52.325161+0000","last_unstale":"2024-03-24T11:34:08.269270+0000","last_undegraded":"2024-03-24T11:34:08.269270+0000","last_fullsized":"2024-03-24T11:34:08.269270+0000","mapping_epoch":267,"log_start":"275'1989","ondisk_log_start":"275'1989","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:16.776651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378732544,"num_objects":5779,"num_object_clones":0,"num_object_copies":17337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5779,"num_write_kb":369856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":524,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.440794+0000","last_change":"2024-03-24T11:32:39.715836+0000","last_active":"2024-03-24T11:34:06.440794+0000","last_peered":"2024-03-24T11:34:06.440794+0000","last_clean":"2024-03-24T11:34:06.440794+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:34:06.440794+0000","last_undegraded":"2024-03-24T11:34:06.440794+0000","last_fullsized":"2024-03-24T11:34:06.440794+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.715759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"213'30","reported_seq":554,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.454873+0000","last_change":"2024-03-24T11:31:31.099444+0000","last_active":"2024-03-24T11:34:06.454873+0000","last_peered":"2024-03-24T11:34:06.454873+0000","last_clean":"2024-03-24T11:34:06.454873+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:34:06.454873+0000","last_undegraded":"2024-03-24T11:34:06.454873+0000","last_fullsized":"2024-03-24T11:34:06.454873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:31:31.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.099209+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":516,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.458195+0000","last_change":"2024-03-24T11:33:36.491136+0000","last_active":"2024-03-24T11:34:06.458195+0000","last_peered":"2024-03-24T11:34:06.458195+0000","last_clean":"2024-03-24T11:34:06.458195+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:34:06.458195+0000","last_undegraded":"2024-03-24T11:34:06.458195+0000","last_fullsized":"2024-03-24T11:34:06.458195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:33:36.491006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:40.856547+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":512,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.440901+0000","last_change":"2024-03-24T11:33:30.982889+0000","last_active":"2024-03-24T11:34:06.440901+0000","last_peered":"2024-03-24T11:34:06.440901+0000","last_clean":"2024-03-24T11:34:06.440901+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:34:06.440901+0000","last_undegraded":"2024-03-24T11:34:06.440901+0000","last_fullsized":"2024-03-24T11:34:06.440901+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:30.982770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:33:30.982770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:53.916965+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":670,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441160+0000","last_change":"2024-03-24T11:31:25.692917+0000","last_active":"2024-03-24T11:34:06.441160+0000","last_peered":"2024-03-24T11:34:06.441160+0000","last_clean":"2024-03-24T11:34:06.441160+0000","last_became_active":"2024-03-24T11:28:38.173086+0000","last_became_peered":"2024-03-24T11:28:38.173086+0000","last_unstale":"2024-03-24T11:34:06.441160+0000","last_undegraded":"2024-03-24T11:34:06.441160+0000","last_fullsized":"2024-03-24T11:34:06.441160+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.692790+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:28:38.972499+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.692790+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"295'5713","reported_seq":5781,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.395858+0000","last_change":"2024-03-24T11:32:52.673849+0000","last_active":"2024-03-24T11:34:08.395858+0000","last_peered":"2024-03-24T11:34:08.395858+0000","last_clean":"2024-03-24T11:34:08.395858+0000","last_became_active":"2024-03-24T11:32:52.673505+0000","last_became_peered":"2024-03-24T11:32:52.673505+0000","last_unstale":"2024-03-24T11:34:08.395858+0000","last_undegraded":"2024-03-24T11:34:08.395858+0000","last_fullsized":"2024-03-24T11:34:08.395858+0000","mapping_epoch":267,"log_start":"272'800","ondisk_log_start":"272'800","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374407168,"num_objects":5713,"num_object_clones":0,"num_object_copies":17139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":365632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":692,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.455038+0000","last_change":"2024-03-24T11:31:08.290147+0000","last_active":"2024-03-24T11:34:06.455038+0000","last_peered":"2024-03-24T11:34:06.455038+0000","last_clean":"2024-03-24T11:34:06.455038+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:34:06.455038+0000","last_undegraded":"2024-03-24T11:34:06.455038+0000","last_fullsized":"2024-03-24T11:34:06.455038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:08.289960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.289960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"213'28","reported_seq":562,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441662+0000","last_change":"2024-03-24T11:32:34.830458+0000","last_active":"2024-03-24T11:34:06.441662+0000","last_peered":"2024-03-24T11:34:06.441662+0000","last_clean":"2024-03-24T11:34:06.441662+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:34:06.441662+0000","last_undegraded":"2024-03-24T11:34:06.441662+0000","last_fullsized":"2024-03-24T11:34:06.441662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:32:34.830322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.830322+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":533,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.452507+0000","last_change":"2024-03-24T11:32:40.035107+0000","last_active":"2024-03-24T11:34:06.452507+0000","last_peered":"2024-03-24T11:34:06.452507+0000","last_clean":"2024-03-24T11:34:06.452507+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:34:06.452507+0000","last_undegraded":"2024-03-24T11:34:06.452507+0000","last_fullsized":"2024-03-24T11:34:06.452507+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:32:40.035022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.035022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":143,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445231+0000","last_change":"2024-03-24T11:32:33.957761+0000","last_active":"2024-03-24T11:34:06.445231+0000","last_peered":"2024-03-24T11:34:06.445231+0000","last_clean":"2024-03-24T11:34:06.445231+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:34:06.445231+0000","last_undegraded":"2024-03-24T11:34:06.445231+0000","last_fullsized":"2024-03-24T11:34:06.445231+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:33.957632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.957632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":662,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.458498+0000","last_change":"2024-03-24T11:31:18.144752+0000","last_active":"2024-03-24T11:34:06.458498+0000","last_peered":"2024-03-24T11:34:06.458498+0000","last_clean":"2024-03-24T11:34:06.458498+0000","last_became_active":"2024-03-24T11:28:38.174888+0000","last_became_peered":"2024-03-24T11:28:38.174888+0000","last_unstale":"2024-03-24T11:34:06.458498+0000","last_undegraded":"2024-03-24T11:34:06.458498+0000","last_fullsized":"2024-03-24T11:34:06.458498+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:18.144322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:18.144322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"295'5731","reported_seq":5800,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.274819+0000","last_change":"2024-03-24T11:32:52.670936+0000","last_active":"2024-03-24T11:34:08.274819+0000","last_peered":"2024-03-24T11:34:08.274819+0000","last_clean":"2024-03-24T11:34:08.274819+0000","last_became_active":"2024-03-24T11:32:52.670591+0000","last_became_peered":"2024-03-24T11:32:52.670591+0000","last_unstale":"2024-03-24T11:34:08.274819+0000","last_undegraded":"2024-03-24T11:34:08.274819+0000","last_fullsized":"2024-03-24T11:34:08.274819+0000","mapping_epoch":267,"log_start":"275'1958","ondisk_log_start":"275'1958","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":3773,"log_dups_size":0,"ondisk_log_size":3773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:20.310931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375586816,"num_objects":5731,"num_object_clones":0,"num_object_copies":17193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":366784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":717,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.450122+0000","last_change":"2024-03-24T11:32:42.033121+0000","last_active":"2024-03-24T11:34:06.450122+0000","last_peered":"2024-03-24T11:34:06.450122+0000","last_clean":"2024-03-24T11:34:06.450122+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:34:06.450122+0000","last_undegraded":"2024-03-24T11:34:06.450122+0000","last_fullsized":"2024-03-24T11:34:06.450122+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.033019+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"213'20","reported_seq":566,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437539+0000","last_change":"2024-03-24T11:32:21.588381+0000","last_active":"2024-03-24T11:34:06.437539+0000","last_peered":"2024-03-24T11:34:06.437539+0000","last_clean":"2024-03-24T11:34:06.437539+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:34:06.437539+0000","last_undegraded":"2024-03-24T11:34:06.437539+0000","last_fullsized":"2024-03-24T11:34:06.437539+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:21.588239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.588239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":543,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438086+0000","last_change":"2024-03-24T11:33:42.448796+0000","last_active":"2024-03-24T11:34:06.438086+0000","last_peered":"2024-03-24T11:34:06.438086+0000","last_clean":"2024-03-24T11:34:06.438086+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:34:06.438086+0000","last_undegraded":"2024-03-24T11:34:06.438086+0000","last_fullsized":"2024-03-24T11:34:06.438086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.448705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:46.343036+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":513,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449558+0000","last_change":"2024-03-24T11:32:22.994060+0000","last_active":"2024-03-24T11:34:06.449558+0000","last_peered":"2024-03-24T11:34:06.449558+0000","last_clean":"2024-03-24T11:34:06.449558+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:34:06.449558+0000","last_undegraded":"2024-03-24T11:34:06.449558+0000","last_fullsized":"2024-03-24T11:34:06.449558+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:32:22.993851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:22.993851+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":692,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.446007+0000","last_change":"2024-03-24T11:33:34.754668+0000","last_active":"2024-03-24T11:34:06.446007+0000","last_peered":"2024-03-24T11:34:06.446007+0000","last_clean":"2024-03-24T11:34:06.446007+0000","last_became_active":"2024-03-24T11:28:38.178260+0000","last_became_peered":"2024-03-24T11:28:38.178260+0000","last_unstale":"2024-03-24T11:34:06.446007+0000","last_undegraded":"2024-03-24T11:34:06.446007+0000","last_fullsized":"2024-03-24T11:34:06.446007+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:34.754561+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:33:34.754561+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:57.926183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"295'5756","reported_seq":5825,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.292519+0000","last_change":"2024-03-24T11:32:52.327929+0000","last_active":"2024-03-24T11:34:08.292519+0000","last_peered":"2024-03-24T11:34:08.292519+0000","last_clean":"2024-03-24T11:34:08.292519+0000","last_became_active":"2024-03-24T11:32:52.327652+0000","last_became_peered":"2024-03-24T11:32:52.327652+0000","last_unstale":"2024-03-24T11:34:08.292519+0000","last_undegraded":"2024-03-24T11:34:08.292519+0000","last_fullsized":"2024-03-24T11:34:08.292519+0000","mapping_epoch":267,"log_start":"272'900","ondisk_log_start":"272'900","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377225216,"num_objects":5756,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5756,"num_write_kb":368384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":548,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445841+0000","last_change":"2024-03-24T11:33:33.804488+0000","last_active":"2024-03-24T11:34:06.445841+0000","last_peered":"2024-03-24T11:34:06.445841+0000","last_clean":"2024-03-24T11:34:06.445841+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:34:06.445841+0000","last_undegraded":"2024-03-24T11:34:06.445841+0000","last_fullsized":"2024-03-24T11:34:06.445841+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:33.804384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.804384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:37.739490+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"213'29","reported_seq":571,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449725+0000","last_change":"2024-03-24T11:32:26.907881+0000","last_active":"2024-03-24T11:34:06.449725+0000","last_peered":"2024-03-24T11:34:06.449725+0000","last_clean":"2024-03-24T11:34:06.449725+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:34:06.449725+0000","last_undegraded":"2024-03-24T11:34:06.449725+0000","last_fullsized":"2024-03-24T11:34:06.449725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:32:26.907769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.907769+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":548,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437385+0000","last_change":"2024-03-24T11:33:20.238699+0000","last_active":"2024-03-24T11:34:06.437385+0000","last_peered":"2024-03-24T11:34:06.437385+0000","last_clean":"2024-03-24T11:34:06.437385+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:34:06.437385+0000","last_undegraded":"2024-03-24T11:34:06.437385+0000","last_fullsized":"2024-03-24T11:34:06.437385+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:20.238585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.238585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:35.247654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":528,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.436482+0000","last_change":"2024-03-24T11:32:45.851666+0000","last_active":"2024-03-24T11:34:06.436482+0000","last_peered":"2024-03-24T11:34:06.436482+0000","last_clean":"2024-03-24T11:34:06.436482+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:34:06.436482+0000","last_undegraded":"2024-03-24T11:34:06.436482+0000","last_fullsized":"2024-03-24T11:34:06.436482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.851566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.851566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":692,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.446107+0000","last_change":"2024-03-24T11:33:32.751957+0000","last_active":"2024-03-24T11:34:06.446107+0000","last_peered":"2024-03-24T11:34:06.446107+0000","last_clean":"2024-03-24T11:34:06.446107+0000","last_became_active":"2024-03-24T11:28:38.174742+0000","last_became_peered":"2024-03-24T11:28:38.174742+0000","last_unstale":"2024-03-24T11:34:06.446107+0000","last_undegraded":"2024-03-24T11:34:06.446107+0000","last_fullsized":"2024-03-24T11:34:06.446107+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:32.751865+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:33:32.751865+0000","last_clean_scrub_stamp":"2024-03-24T11:33:32.751865+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:51.954315+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"295'5384","reported_seq":5459,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:09.801581+0000","last_change":"2024-03-24T11:33:46.306625+0000","last_active":"2024-03-24T11:34:09.801581+0000","last_peered":"2024-03-24T11:34:09.801581+0000","last_clean":"2024-03-24T11:34:09.801581+0000","last_became_active":"2024-03-24T11:33:46.285778+0000","last_became_peered":"2024-03-24T11:33:46.285778+0000","last_unstale":"2024-03-24T11:34:09.801581+0000","last_undegraded":"2024-03-24T11:34:09.801581+0000","last_fullsized":"2024-03-24T11:34:09.801581+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:10.063665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193855488,"num_objects":2958,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2958,"num_write_kb":189312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":523,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438611+0000","last_change":"2024-03-24T11:33:35.552752+0000","last_active":"2024-03-24T11:34:06.438611+0000","last_peered":"2024-03-24T11:34:06.438611+0000","last_clean":"2024-03-24T11:34:06.438611+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:34:06.438611+0000","last_undegraded":"2024-03-24T11:34:06.438611+0000","last_fullsized":"2024-03-24T11:34:06.438611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:35.552605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:33:35.552605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:44.893540+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"213'24","reported_seq":73,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438166+0000","last_change":"2024-03-24T11:33:10.483025+0000","last_active":"2024-03-24T11:34:06.438166+0000","last_peered":"2024-03-24T11:34:06.438166+0000","last_clean":"2024-03-24T11:34:06.438166+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:34:06.438166+0000","last_undegraded":"2024-03-24T11:34:06.438166+0000","last_fullsized":"2024-03-24T11:34:06.438166+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:21.972644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.972644+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":400,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.035598+0000","last_change":"2024-03-24T11:32:39.822026+0000","last_active":"2024-03-24T11:34:08.035598+0000","last_peered":"2024-03-24T11:34:08.035598+0000","last_clean":"2024-03-24T11:34:08.035598+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:34:08.035598+0000","last_undegraded":"2024-03-24T11:34:08.035598+0000","last_fullsized":"2024-03-24T11:34:08.035598+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:32:39.821933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.821933+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":535,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.448832+0000","last_change":"2024-03-24T11:33:19.182254+0000","last_active":"2024-03-24T11:34:06.448832+0000","last_peered":"2024-03-24T11:34:06.448832+0000","last_clean":"2024-03-24T11:34:06.448832+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:34:06.448832+0000","last_undegraded":"2024-03-24T11:34:06.448832+0000","last_fullsized":"2024-03-24T11:34:06.448832+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.182060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.182060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:28.529107+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":536,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437187+0000","last_change":"2024-03-24T11:33:42.297388+0000","last_active":"2024-03-24T11:34:06.437187+0000","last_peered":"2024-03-24T11:34:06.437187+0000","last_clean":"2024-03-24T11:34:06.437187+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:34:06.437187+0000","last_undegraded":"2024-03-24T11:34:06.437187+0000","last_fullsized":"2024-03-24T11:34:06.437187+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.297267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.297267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:01.314016+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":678,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.439235+0000","last_change":"2024-03-24T11:32:26.629982+0000","last_active":"2024-03-24T11:34:06.439235+0000","last_peered":"2024-03-24T11:34:06.439235+0000","last_clean":"2024-03-24T11:34:06.439235+0000","last_became_active":"2024-03-24T11:28:38.173843+0000","last_became_peered":"2024-03-24T11:28:38.173843+0000","last_unstale":"2024-03-24T11:34:06.439235+0000","last_undegraded":"2024-03-24T11:34:06.439235+0000","last_fullsized":"2024-03-24T11:34:06.439235+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:26.629894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.629894+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"295'5686","reported_seq":5755,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.409701+0000","last_change":"2024-03-24T11:32:52.328051+0000","last_active":"2024-03-24T11:34:08.409701+0000","last_peered":"2024-03-24T11:34:08.409701+0000","last_clean":"2024-03-24T11:34:08.409701+0000","last_became_active":"2024-03-24T11:32:52.327768+0000","last_became_peered":"2024-03-24T11:32:52.327768+0000","last_unstale":"2024-03-24T11:34:08.409701+0000","last_undegraded":"2024-03-24T11:34:08.409701+0000","last_fullsized":"2024-03-24T11:34:08.409701+0000","mapping_epoch":267,"log_start":"272'800","ondisk_log_start":"272'800","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372637696,"num_objects":5686,"num_object_clones":0,"num_object_copies":17058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5686,"num_write_kb":363904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":143,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.444143+0000","last_change":"2024-03-24T11:32:41.207872+0000","last_active":"2024-03-24T11:34:06.444143+0000","last_peered":"2024-03-24T11:34:06.444143+0000","last_clean":"2024-03-24T11:34:06.444143+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:34:06.444143+0000","last_undegraded":"2024-03-24T11:34:06.444143+0000","last_fullsized":"2024-03-24T11:34:06.444143+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:41.207792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.207792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"213'36","reported_seq":783,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445591+0000","last_change":"2024-03-24T11:33:42.837753+0000","last_active":"2024-03-24T11:34:06.445591+0000","last_peered":"2024-03-24T11:34:06.445591+0000","last_clean":"2024-03-24T11:34:06.445591+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:34:06.445591+0000","last_undegraded":"2024-03-24T11:34:06.445591+0000","last_fullsized":"2024-03-24T11:34:06.445591+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'36","last_scrub_stamp":"2024-03-24T11:33:42.837610+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.837610+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:04.268624+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":522,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.455411+0000","last_change":"2024-03-24T11:32:28.089047+0000","last_active":"2024-03-24T11:34:06.455411+0000","last_peered":"2024-03-24T11:34:06.455411+0000","last_clean":"2024-03-24T11:34:06.455411+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:34:06.455411+0000","last_undegraded":"2024-03-24T11:34:06.455411+0000","last_fullsized":"2024-03-24T11:34:06.455411+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.088969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.088969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":439,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449491+0000","last_change":"2024-03-24T11:33:18.225225+0000","last_active":"2024-03-24T11:34:06.449491+0000","last_peered":"2024-03-24T11:34:06.449491+0000","last_clean":"2024-03-24T11:34:06.449491+0000","last_became_active":"2024-03-24T11:30:35.355009+0000","last_became_peered":"2024-03-24T11:30:35.355009+0000","last_unstale":"2024-03-24T11:34:06.449491+0000","last_undegraded":"2024-03-24T11:34:06.449491+0000","last_fullsized":"2024-03-24T11:34:06.449491+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:18.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:18.225111+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:31.427035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":527,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438446+0000","last_change":"2024-03-24T11:31:26.676287+0000","last_active":"2024-03-24T11:34:06.438446+0000","last_peered":"2024-03-24T11:34:06.438446+0000","last_clean":"2024-03-24T11:34:06.438446+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:34:06.438446+0000","last_undegraded":"2024-03-24T11:34:06.438446+0000","last_fullsized":"2024-03-24T11:34:06.438446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:26.676209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:31:26.676209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"295'5287","reported_seq":5362,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.524734+0000","last_change":"2024-03-24T11:33:46.282181+0000","last_active":"2024-03-24T11:34:08.524734+0000","last_peered":"2024-03-24T11:34:08.524734+0000","last_clean":"2024-03-24T11:34:08.524734+0000","last_became_active":"2024-03-24T11:33:46.279697+0000","last_became_peered":"2024-03-24T11:33:46.279697+0000","last_unstale":"2024-03-24T11:34:08.524734+0000","last_undegraded":"2024-03-24T11:34:08.524734+0000","last_fullsized":"2024-03-24T11:34:08.524734+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5287,"log_dups_size":0,"ondisk_log_size":5287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:09.420070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189071360,"num_objects":2885,"num_object_clones":0,"num_object_copies":8655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":184640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":540,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445441+0000","last_change":"2024-03-24T11:33:24.909166+0000","last_active":"2024-03-24T11:34:06.445441+0000","last_peered":"2024-03-24T11:34:06.445441+0000","last_clean":"2024-03-24T11:34:06.445441+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:34:06.445441+0000","last_undegraded":"2024-03-24T11:34:06.445441+0000","last_fullsized":"2024-03-24T11:34:06.445441+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:24.908806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.908806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:44.157222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"213'17","reported_seq":543,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445251+0000","last_change":"2024-03-24T11:32:38.137085+0000","last_active":"2024-03-24T11:34:06.445251+0000","last_peered":"2024-03-24T11:34:06.445251+0000","last_clean":"2024-03-24T11:34:06.445251+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:34:06.445251+0000","last_undegraded":"2024-03-24T11:34:06.445251+0000","last_fullsized":"2024-03-24T11:34:06.445251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'17","last_scrub_stamp":"2024-03-24T11:32:38.136966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.136966+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":557,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442791+0000","last_change":"2024-03-24T11:33:29.862071+0000","last_active":"2024-03-24T11:34:06.442791+0000","last_peered":"2024-03-24T11:34:06.442791+0000","last_clean":"2024-03-24T11:34:06.442791+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:34:06.442791+0000","last_undegraded":"2024-03-24T11:34:06.442791+0000","last_fullsized":"2024-03-24T11:34:06.442791+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:29.861850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:33:29.861850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:52.996845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":548,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.440458+0000","last_change":"2024-03-24T11:33:37.369520+0000","last_active":"2024-03-24T11:34:06.440458+0000","last_peered":"2024-03-24T11:34:06.440458+0000","last_clean":"2024-03-24T11:34:06.440458+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:34:06.440458+0000","last_undegraded":"2024-03-24T11:34:06.440458+0000","last_fullsized":"2024-03-24T11:34:06.440458+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:37.369254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:37.369254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:43.784548+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":143,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445654+0000","last_change":"2024-03-24T11:32:42.994801+0000","last_active":"2024-03-24T11:34:06.445654+0000","last_peered":"2024-03-24T11:34:06.445654+0000","last_clean":"2024-03-24T11:34:06.445654+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:34:06.445654+0000","last_undegraded":"2024-03-24T11:34:06.445654+0000","last_fullsized":"2024-03-24T11:34:06.445654+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.994694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.994694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":439,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.465589+0000","last_change":"2024-03-24T11:33:20.172450+0000","last_active":"2024-03-24T11:34:06.465589+0000","last_peered":"2024-03-24T11:34:06.465589+0000","last_clean":"2024-03-24T11:34:06.465589+0000","last_became_active":"2024-03-24T11:30:35.356166+0000","last_became_peered":"2024-03-24T11:30:35.356166+0000","last_unstale":"2024-03-24T11:34:06.465589+0000","last_undegraded":"2024-03-24T11:34:06.465589+0000","last_fullsized":"2024-03-24T11:34:06.465589+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:20.172032+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.172032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:41.569472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"295'5750","reported_seq":5817,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.277084+0000","last_change":"2024-03-24T11:32:52.326526+0000","last_active":"2024-03-24T11:34:08.277084+0000","last_peered":"2024-03-24T11:34:08.277084+0000","last_clean":"2024-03-24T11:34:08.277084+0000","last_became_active":"2024-03-24T11:32:52.326215+0000","last_became_peered":"2024-03-24T11:32:52.326215+0000","last_unstale":"2024-03-24T11:34:08.277084+0000","last_undegraded":"2024-03-24T11:34:08.277084+0000","last_fullsized":"2024-03-24T11:34:08.277084+0000","mapping_epoch":267,"log_start":"272'900","ondisk_log_start":"272'900","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:11.644980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376832000,"num_objects":5750,"num_object_clones":0,"num_object_copies":17250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5750,"num_write_kb":368000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":524,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449505+0000","last_change":"2024-03-24T11:33:33.162620+0000","last_active":"2024-03-24T11:34:06.449505+0000","last_peered":"2024-03-24T11:34:06.449505+0000","last_clean":"2024-03-24T11:34:06.449505+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:34:06.449505+0000","last_undegraded":"2024-03-24T11:34:06.449505+0000","last_fullsized":"2024-03-24T11:34:06.449505+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:33.162392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.162392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:01.118507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"213'40","reported_seq":786,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437507+0000","last_change":"2024-03-24T11:32:43.782374+0000","last_active":"2024-03-24T11:34:06.437507+0000","last_peered":"2024-03-24T11:34:06.437507+0000","last_clean":"2024-03-24T11:34:06.437507+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:34:06.437507+0000","last_undegraded":"2024-03-24T11:34:06.437507+0000","last_fullsized":"2024-03-24T11:34:06.437507+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'40","last_scrub_stamp":"2024-03-24T11:32:43.782256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.782256+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":524,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441011+0000","last_change":"2024-03-24T11:31:05.775644+0000","last_active":"2024-03-24T11:34:06.441011+0000","last_peered":"2024-03-24T11:34:06.441011+0000","last_clean":"2024-03-24T11:34:06.441011+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:34:06.441011+0000","last_undegraded":"2024-03-24T11:34:06.441011+0000","last_fullsized":"2024-03-24T11:34:06.441011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:05.775534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:05.775534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":543,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437830+0000","last_change":"2024-03-24T11:32:28.707278+0000","last_active":"2024-03-24T11:34:06.437830+0000","last_peered":"2024-03-24T11:34:06.437830+0000","last_clean":"2024-03-24T11:34:06.437830+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:34:06.437830+0000","last_undegraded":"2024-03-24T11:34:06.437830+0000","last_fullsized":"2024-03-24T11:34:06.437830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:32:28.707184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.707184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":504,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.456666+0000","last_change":"2024-03-24T11:33:35.480861+0000","last_active":"2024-03-24T11:34:06.456666+0000","last_peered":"2024-03-24T11:34:06.456666+0000","last_clean":"2024-03-24T11:34:06.456666+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:34:06.456666+0000","last_undegraded":"2024-03-24T11:34:06.456666+0000","last_fullsized":"2024-03-24T11:34:06.456666+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:35.480687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:35.480687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:01.069305+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":678,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437541+0000","last_change":"2024-03-24T11:32:29.691475+0000","last_active":"2024-03-24T11:34:06.437541+0000","last_peered":"2024-03-24T11:34:06.437541+0000","last_clean":"2024-03-24T11:34:06.437541+0000","last_became_active":"2024-03-24T11:28:38.173832+0000","last_became_peered":"2024-03-24T11:28:38.173832+0000","last_unstale":"2024-03-24T11:34:06.437541+0000","last_undegraded":"2024-03-24T11:34:06.437541+0000","last_fullsized":"2024-03-24T11:34:06.437541+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:29.691383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:29.691383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"295'5265","reported_seq":5340,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.638376+0000","last_change":"2024-03-24T11:33:46.317330+0000","last_active":"2024-03-24T11:34:08.638376+0000","last_peered":"2024-03-24T11:34:08.638376+0000","last_clean":"2024-03-24T11:34:08.638376+0000","last_became_active":"2024-03-24T11:33:46.315078+0000","last_became_peered":"2024-03-24T11:33:46.315078+0000","last_unstale":"2024-03-24T11:34:08.638376+0000","last_undegraded":"2024-03-24T11:34:08.638376+0000","last_fullsized":"2024-03-24T11:34:08.638376+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:10.011686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188514304,"num_objects":2876,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2876,"num_write_kb":184096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"213'15","reported_seq":552,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449991+0000","last_change":"2024-03-24T11:32:37.060211+0000","last_active":"2024-03-24T11:34:06.449991+0000","last_peered":"2024-03-24T11:34:06.449991+0000","last_clean":"2024-03-24T11:34:06.449991+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:34:06.449991+0000","last_undegraded":"2024-03-24T11:34:06.449991+0000","last_fullsized":"2024-03-24T11:34:06.449991+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:32:37.060067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.060067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":537,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438030+0000","last_change":"2024-03-24T11:31:20.465180+0000","last_active":"2024-03-24T11:34:06.438030+0000","last_peered":"2024-03-24T11:34:06.438030+0000","last_clean":"2024-03-24T11:34:06.438030+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:34:06.438030+0000","last_undegraded":"2024-03-24T11:34:06.438030+0000","last_fullsized":"2024-03-24T11:34:06.438030+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:31:20.465096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:20.465096+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":539,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.443191+0000","last_change":"2024-03-24T11:33:37.859003+0000","last_active":"2024-03-24T11:34:06.443191+0000","last_peered":"2024-03-24T11:34:06.443191+0000","last_clean":"2024-03-24T11:34:06.443191+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:34:06.443191+0000","last_undegraded":"2024-03-24T11:34:06.443191+0000","last_fullsized":"2024-03-24T11:34:06.443191+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:37.858877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:37.858877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:56.452752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449081+0000","last_change":"2024-03-24T11:33:37.314906+0000","last_active":"2024-03-24T11:34:06.449081+0000","last_peered":"2024-03-24T11:34:06.449081+0000","last_clean":"2024-03-24T11:34:06.449081+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:34:06.449081+0000","last_undegraded":"2024-03-24T11:34:06.449081+0000","last_fullsized":"2024-03-24T11:34:06.449081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:37.314690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:37.314690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:39.761086+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":509,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441255+0000","last_change":"2024-03-24T11:31:33.687541+0000","last_active":"2024-03-24T11:34:06.441255+0000","last_peered":"2024-03-24T11:34:06.441255+0000","last_clean":"2024-03-24T11:34:06.441255+0000","last_became_active":"2024-03-24T11:28:43.192246+0000","last_became_peered":"2024-03-24T11:28:43.192246+0000","last_unstale":"2024-03-24T11:34:06.441255+0000","last_undegraded":"2024-03-24T11:34:06.441255+0000","last_fullsized":"2024-03-24T11:34:06.441255+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:33.687261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:33.687261+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"295'5306","reported_seq":5384,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:09.788702+0000","last_change":"2024-03-24T11:33:46.328543+0000","last_active":"2024-03-24T11:34:09.788702+0000","last_peered":"2024-03-24T11:34:09.788702+0000","last_clean":"2024-03-24T11:34:09.788702+0000","last_became_active":"2024-03-24T11:33:46.308480+0000","last_became_peered":"2024-03-24T11:33:46.308480+0000","last_unstale":"2024-03-24T11:34:09.788702+0000","last_undegraded":"2024-03-24T11:34:09.788702+0000","last_fullsized":"2024-03-24T11:34:09.788702+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:10.063691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191365120,"num_objects":2920,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":186880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"213'44","reported_seq":608,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445780+0000","last_change":"2024-03-24T11:33:24.826729+0000","last_active":"2024-03-24T11:34:06.445780+0000","last_peered":"2024-03-24T11:34:06.445780+0000","last_clean":"2024-03-24T11:34:06.445780+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:34:06.445780+0000","last_undegraded":"2024-03-24T11:34:06.445780+0000","last_fullsized":"2024-03-24T11:34:06.445780+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:33:24.826519+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.826519+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:31.128314+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438203+0000","last_change":"2024-03-24T11:32:38.827248+0000","last_active":"2024-03-24T11:34:06.438203+0000","last_peered":"2024-03-24T11:34:06.438203+0000","last_clean":"2024-03-24T11:34:06.438203+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:34:06.438203+0000","last_undegraded":"2024-03-24T11:34:06.438203+0000","last_fullsized":"2024-03-24T11:34:06.438203+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.827167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.827167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":539,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445605+0000","last_change":"2024-03-24T11:33:27.894256+0000","last_active":"2024-03-24T11:34:06.445605+0000","last_peered":"2024-03-24T11:34:06.445605+0000","last_clean":"2024-03-24T11:34:06.445605+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:34:06.445605+0000","last_undegraded":"2024-03-24T11:34:06.445605+0000","last_fullsized":"2024-03-24T11:34:06.445605+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:33:27.894028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:38.736859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":143,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442163+0000","last_change":"2024-03-24T11:33:22.957967+0000","last_active":"2024-03-24T11:34:06.442163+0000","last_peered":"2024-03-24T11:34:06.442163+0000","last_clean":"2024-03-24T11:34:06.442163+0000","last_became_active":"2024-03-24T11:30:35.355210+0000","last_became_peered":"2024-03-24T11:30:35.355210+0000","last_unstale":"2024-03-24T11:34:06.442163+0000","last_undegraded":"2024-03-24T11:34:06.442163+0000","last_fullsized":"2024-03-24T11:34:06.442163+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:22.957812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.957812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:38.348467+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":691,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445836+0000","last_change":"2024-03-24T11:32:34.973136+0000","last_active":"2024-03-24T11:34:06.445836+0000","last_peered":"2024-03-24T11:34:06.445836+0000","last_clean":"2024-03-24T11:34:06.445836+0000","last_became_active":"2024-03-24T11:28:43.191874+0000","last_became_peered":"2024-03-24T11:28:43.191874+0000","last_unstale":"2024-03-24T11:34:06.445836+0000","last_undegraded":"2024-03-24T11:34:06.445836+0000","last_fullsized":"2024-03-24T11:34:06.445836+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:34.973008+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.973008+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"295'5178","reported_seq":5255,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.381872+0000","last_change":"2024-03-24T11:33:46.308282+0000","last_active":"2024-03-24T11:34:08.381872+0000","last_peered":"2024-03-24T11:34:08.381872+0000","last_clean":"2024-03-24T11:34:08.381872+0000","last_became_active":"2024-03-24T11:33:46.296101+0000","last_became_peered":"2024-03-24T11:33:46.296101+0000","last_unstale":"2024-03-24T11:34:08.381872+0000","last_undegraded":"2024-03-24T11:34:08.381872+0000","last_fullsized":"2024-03-24T11:34:08.381872+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:12.885525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184975360,"num_objects":2822,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2822,"num_write_kb":180640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":540,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442009+0000","last_change":"2024-03-24T11:32:44.207912+0000","last_active":"2024-03-24T11:34:06.442009+0000","last_peered":"2024-03-24T11:34:06.442009+0000","last_clean":"2024-03-24T11:34:06.442009+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:34:06.442009+0000","last_undegraded":"2024-03-24T11:34:06.442009+0000","last_fullsized":"2024-03-24T11:34:06.442009+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:44.207817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:44.207817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"213'18","reported_seq":552,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.444138+0000","last_change":"2024-03-24T11:31:11.798229+0000","last_active":"2024-03-24T11:34:06.444138+0000","last_peered":"2024-03-24T11:34:06.444138+0000","last_clean":"2024-03-24T11:34:06.444138+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:34:06.444138+0000","last_undegraded":"2024-03-24T11:34:06.444138+0000","last_fullsized":"2024-03-24T11:34:06.444138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.798089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.798089+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441424+0000","last_change":"2024-03-24T11:33:20.771559+0000","last_active":"2024-03-24T11:34:06.441424+0000","last_peered":"2024-03-24T11:34:06.441424+0000","last_clean":"2024-03-24T11:34:06.441424+0000","last_became_active":"2024-03-24T11:30:35.356055+0000","last_became_peered":"2024-03-24T11:30:35.356055+0000","last_unstale":"2024-03-24T11:34:06.441424+0000","last_undegraded":"2024-03-24T11:34:06.441424+0000","last_fullsized":"2024-03-24T11:34:06.441424+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:20.771417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.771417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:40.688846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":503,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.456163+0000","last_change":"2024-03-24T11:33:42.747143+0000","last_active":"2024-03-24T11:34:06.456163+0000","last_peered":"2024-03-24T11:34:06.456163+0000","last_clean":"2024-03-24T11:34:06.456163+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:34:06.456163+0000","last_undegraded":"2024-03-24T11:34:06.456163+0000","last_fullsized":"2024-03-24T11:34:06.456163+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.746752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.746752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:10.359810+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":675,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.457961+0000","last_change":"2024-03-24T11:33:21.194045+0000","last_active":"2024-03-24T11:34:06.457961+0000","last_peered":"2024-03-24T11:34:06.457961+0000","last_clean":"2024-03-24T11:34:06.457961+0000","last_became_active":"2024-03-24T11:28:43.189494+0000","last_became_peered":"2024-03-24T11:28:43.189494+0000","last_unstale":"2024-03-24T11:34:06.457961+0000","last_undegraded":"2024-03-24T11:34:06.457961+0000","last_fullsized":"2024-03-24T11:34:06.457961+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:21.193899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:21.193899+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:25.212189+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"295'5309","reported_seq":5387,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.525606+0000","last_change":"2024-03-24T11:33:46.331122+0000","last_active":"2024-03-24T11:34:08.525606+0000","last_peered":"2024-03-24T11:34:08.525606+0000","last_clean":"2024-03-24T11:34:08.525606+0000","last_became_active":"2024-03-24T11:33:46.315487+0000","last_became_peered":"2024-03-24T11:33:46.315487+0000","last_unstale":"2024-03-24T11:34:08.525606+0000","last_undegraded":"2024-03-24T11:34:08.525606+0000","last_fullsized":"2024-03-24T11:34:08.525606+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:09.420094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191430656,"num_objects":2921,"num_object_clones":0,"num_object_copies":8763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2921,"num_write_kb":186944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"238'9","reported_seq":211,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.444878+0000","last_change":"2024-03-24T11:32:43.964046+0000","last_active":"2024-03-24T11:34:06.444878+0000","last_peered":"2024-03-24T11:34:06.444878+0000","last_clean":"2024-03-24T11:34:06.444878+0000","last_became_active":"2024-03-24T11:30:35.359635+0000","last_became_peered":"2024-03-24T11:30:35.359635+0000","last_unstale":"2024-03-24T11:34:06.444878+0000","last_undegraded":"2024-03-24T11:34:06.444878+0000","last_fullsized":"2024-03-24T11:34:06.444878+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'9","last_scrub_stamp":"2024-03-24T11:32:43.963946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.963946+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"213'34","reported_seq":578,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437357+0000","last_change":"2024-03-24T11:31:12.435804+0000","last_active":"2024-03-24T11:34:06.437357+0000","last_peered":"2024-03-24T11:34:06.437357+0000","last_clean":"2024-03-24T11:34:06.437357+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:34:06.437357+0000","last_undegraded":"2024-03-24T11:34:06.437357+0000","last_fullsized":"2024-03-24T11:34:06.437357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:31:12.435639+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:31:12.435639+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":337,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442360+0000","last_change":"2024-03-24T11:33:22.820364+0000","last_active":"2024-03-24T11:34:06.442360+0000","last_peered":"2024-03-24T11:34:06.442360+0000","last_clean":"2024-03-24T11:34:06.442360+0000","last_became_active":"2024-03-24T11:30:35.356100+0000","last_became_peered":"2024-03-24T11:30:35.356100+0000","last_unstale":"2024-03-24T11:34:06.442360+0000","last_undegraded":"2024-03-24T11:34:06.442360+0000","last_fullsized":"2024-03-24T11:34:06.442360+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:33:22.820253+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.820253+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:44.928606+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438358+0000","last_change":"2024-03-24T11:31:14.458586+0000","last_active":"2024-03-24T11:34:06.438358+0000","last_peered":"2024-03-24T11:34:06.438358+0000","last_clean":"2024-03-24T11:34:06.438358+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:34:06.438358+0000","last_undegraded":"2024-03-24T11:34:06.438358+0000","last_fullsized":"2024-03-24T11:34:06.438358+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.458470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.458470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":594,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441136+0000","last_change":"2024-03-24T11:31:04.735517+0000","last_active":"2024-03-24T11:34:06.441136+0000","last_peered":"2024-03-24T11:34:06.441136+0000","last_clean":"2024-03-24T11:34:06.441136+0000","last_became_active":"2024-03-24T11:30:35.354637+0000","last_became_peered":"2024-03-24T11:30:35.354637+0000","last_unstale":"2024-03-24T11:34:06.441136+0000","last_undegraded":"2024-03-24T11:34:06.441136+0000","last_fullsized":"2024-03-24T11:34:06.441136+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:04.735433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:04.735433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.19","version":"295'5366","reported_seq":5442,"reported_epoch":295,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-24T11:34:08.805087+0000","last_change":"2024-03-24T11:33:49.740191+0000","last_active":"2024-03-24T11:34:08.805087+0000","last_peered":"2024-03-24T11:34:08.805087+0000","last_clean":"2024-03-24T11:33:47.231317+0000","last_became_active":"2024-03-24T11:33:49.735290+0000","last_became_peered":"2024-03-24T11:33:49.735290+0000","last_unstale":"2024-03-24T11:34:08.805087+0000","last_undegraded":"2024-03-24T11:33:49.650437+0000","last_fullsized":"2024-03-24T11:33:49.285683+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194248704,"num_objects":2964,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4822,"num_objects_misplaced":2411,"num_objects_unfound":0,"num_objects_dirty":2964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":189696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[2,1],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"1,2","objects":553},{"shards":"2,5,7","objects":2411}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"213'15","reported_seq":524,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.458679+0000","last_change":"2024-03-24T11:30:14.150582+0000","last_active":"2024-03-24T11:34:06.458679+0000","last_peered":"2024-03-24T11:34:06.458679+0000","last_clean":"2024-03-24T11:34:06.458679+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:34:06.458679+0000","last_undegraded":"2024-03-24T11:34:06.458679+0000","last_fullsized":"2024-03-24T11:34:06.458679+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:30:14.150454+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:30:14.150454+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":547,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445032+0000","last_change":"2024-03-24T11:32:40.958471+0000","last_active":"2024-03-24T11:34:06.445032+0000","last_peered":"2024-03-24T11:34:06.445032+0000","last_clean":"2024-03-24T11:34:06.445032+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:34:06.445032+0000","last_undegraded":"2024-03-24T11:34:06.445032+0000","last_fullsized":"2024-03-24T11:34:06.445032+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:40.958389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.958389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":523,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438240+0000","last_change":"2024-03-24T11:32:39.835187+0000","last_active":"2024-03-24T11:34:06.438240+0000","last_peered":"2024-03-24T11:34:06.438240+0000","last_clean":"2024-03-24T11:34:06.438240+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:34:06.438240+0000","last_undegraded":"2024-03-24T11:34:06.438240+0000","last_fullsized":"2024-03-24T11:34:06.438240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.835097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.835097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":151,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445731+0000","last_change":"2024-03-24T11:33:36.829894+0000","last_active":"2024-03-24T11:34:06.445731+0000","last_peered":"2024-03-24T11:34:06.445731+0000","last_clean":"2024-03-24T11:34:06.445731+0000","last_became_active":"2024-03-24T11:30:35.353212+0000","last_became_peered":"2024-03-24T11:30:35.353212+0000","last_unstale":"2024-03-24T11:34:06.445731+0000","last_undegraded":"2024-03-24T11:34:06.445731+0000","last_fullsized":"2024-03-24T11:34:06.445731+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:36.829776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:33:36.829776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:06.589530+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":509,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437615+0000","last_change":"2024-03-24T11:31:24.533967+0000","last_active":"2024-03-24T11:34:06.437615+0000","last_peered":"2024-03-24T11:34:06.437615+0000","last_clean":"2024-03-24T11:34:06.437615+0000","last_became_active":"2024-03-24T11:28:43.189410+0000","last_became_peered":"2024-03-24T11:28:43.189410+0000","last_unstale":"2024-03-24T11:34:06.437615+0000","last_undegraded":"2024-03-24T11:34:06.437615+0000","last_fullsized":"2024-03-24T11:34:06.437615+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:24.533874+0000","last_clean_scrub_stamp":"2024-03-24T11:31:24.533874+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.18","version":"295'5332","reported_seq":7010,"reported_epoch":295,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-24T11:34:08.842937+0000","last_change":"2024-03-24T11:33:49.698019+0000","last_active":"2024-03-24T11:34:08.842937+0000","last_peered":"2024-03-24T11:34:08.842937+0000","last_clean":"2024-03-24T11:33:47.261573+0000","last_became_active":"2024-03-24T11:33:49.656747+0000","last_became_peered":"2024-03-24T11:33:49.656747+0000","last_unstale":"2024-03-24T11:34:08.842937+0000","last_undegraded":"2024-03-24T11:34:08.842937+0000","last_fullsized":"2024-03-24T11:33:49.282476+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5332,"log_dups_size":0,"ondisk_log_size":5332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190447616,"num_objects":2906,"num_object_clones":0,"num_object_copies":8718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3878,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2906,"num_write_kb":185984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":34996224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":1939},{"shards":"1,2","objects":967}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":685,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.440914+0000","last_change":"2024-03-24T11:32:33.790867+0000","last_active":"2024-03-24T11:34:06.440914+0000","last_peered":"2024-03-24T11:34:06.440914+0000","last_clean":"2024-03-24T11:34:06.440914+0000","last_became_active":"2024-03-24T11:28:43.192449+0000","last_became_peered":"2024-03-24T11:28:43.192449+0000","last_unstale":"2024-03-24T11:34:06.440914+0000","last_undegraded":"2024-03-24T11:34:06.440914+0000","last_fullsized":"2024-03-24T11:34:06.440914+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:32:33.790777+0000","last_clean_scrub_stamp":"2024-03-24T11:32:33.790777+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"213'19","reported_seq":556,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.452705+0000","last_change":"2024-03-24T11:32:32.023601+0000","last_active":"2024-03-24T11:34:06.452705+0000","last_peered":"2024-03-24T11:34:06.452705+0000","last_clean":"2024-03-24T11:34:06.452705+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:34:06.452705+0000","last_undegraded":"2024-03-24T11:34:06.452705+0000","last_fullsized":"2024-03-24T11:34:06.452705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:32.023502+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.023502+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442131+0000","last_change":"2024-03-24T11:33:33.927309+0000","last_active":"2024-03-24T11:34:06.442131+0000","last_peered":"2024-03-24T11:34:06.442131+0000","last_clean":"2024-03-24T11:34:06.442131+0000","last_became_active":"2024-03-24T11:30:35.358928+0000","last_became_peered":"2024-03-24T11:30:35.358928+0000","last_unstale":"2024-03-24T11:34:06.442131+0000","last_undegraded":"2024-03-24T11:34:06.442131+0000","last_fullsized":"2024-03-24T11:34:06.442131+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:33.927191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.927191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:35.624852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":507,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.458584+0000","last_change":"2024-03-24T11:33:32.456675+0000","last_active":"2024-03-24T11:34:06.458584+0000","last_peered":"2024-03-24T11:34:06.458584+0000","last_clean":"2024-03-24T11:34:06.458584+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:34:06.458584+0000","last_undegraded":"2024-03-24T11:34:06.458584+0000","last_fullsized":"2024-03-24T11:34:06.458584+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:32.456539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:33:32.456539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:51.154252+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":527,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.444056+0000","last_change":"2024-03-24T11:32:26.898659+0000","last_active":"2024-03-24T11:34:06.444056+0000","last_peered":"2024-03-24T11:34:06.444056+0000","last_clean":"2024-03-24T11:34:06.444056+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:34:06.444056+0000","last_undegraded":"2024-03-24T11:34:06.444056+0000","last_fullsized":"2024-03-24T11:34:06.444056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:26.898583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.898583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"213'15","reported_seq":552,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438256+0000","last_change":"2024-03-24T11:31:27.541622+0000","last_active":"2024-03-24T11:34:06.438256+0000","last_peered":"2024-03-24T11:34:06.438256+0000","last_clean":"2024-03-24T11:34:06.438256+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:34:06.438256+0000","last_undegraded":"2024-03-24T11:34:06.438256+0000","last_fullsized":"2024-03-24T11:34:06.438256+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:27.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.541510+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.465491+0000","last_change":"2024-03-24T11:32:45.985820+0000","last_active":"2024-03-24T11:34:06.465491+0000","last_peered":"2024-03-24T11:34:06.465491+0000","last_clean":"2024-03-24T11:34:06.465491+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:34:06.465491+0000","last_undegraded":"2024-03-24T11:34:06.465491+0000","last_fullsized":"2024-03-24T11:34:06.465491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.985741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.985741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:14.491398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":699,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.459184+0000","last_change":"2024-03-24T11:31:13.201069+0000","last_active":"2024-03-24T11:34:06.459184+0000","last_peered":"2024-03-24T11:34:06.459184+0000","last_clean":"2024-03-24T11:34:06.459184+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:34:06.459184+0000","last_undegraded":"2024-03-24T11:34:06.459184+0000","last_fullsized":"2024-03-24T11:34:06.459184+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:13.200503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.200503+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":511,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442513+0000","last_change":"2024-03-24T11:32:27.764304+0000","last_active":"2024-03-24T11:34:06.442513+0000","last_peered":"2024-03-24T11:34:06.442513+0000","last_clean":"2024-03-24T11:34:06.442513+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:34:06.442513+0000","last_undegraded":"2024-03-24T11:34:06.442513+0000","last_fullsized":"2024-03-24T11:34:06.442513+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.764203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.764203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":517,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437464+0000","last_change":"2024-03-24T11:33:39.514047+0000","last_active":"2024-03-24T11:34:06.437464+0000","last_peered":"2024-03-24T11:34:06.437464+0000","last_clean":"2024-03-24T11:34:06.437464+0000","last_became_active":"2024-03-24T11:28:43.188941+0000","last_became_peered":"2024-03-24T11:28:43.188941+0000","last_unstale":"2024-03-24T11:34:06.437464+0000","last_undegraded":"2024-03-24T11:34:06.437464+0000","last_fullsized":"2024-03-24T11:34:06.437464+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:39.513922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:39.513922+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:51.508855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"213'20","reported_seq":541,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.458991+0000","last_change":"2024-03-24T11:33:19.230202+0000","last_active":"2024-03-24T11:34:06.458991+0000","last_peered":"2024-03-24T11:34:06.458991+0000","last_clean":"2024-03-24T11:34:06.458991+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:34:06.458991+0000","last_undegraded":"2024-03-24T11:34:06.458991+0000","last_fullsized":"2024-03-24T11:34:06.458991+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:19.229920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.229920+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:47.588700+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441167+0000","last_change":"2024-03-24T11:31:30.728130+0000","last_active":"2024-03-24T11:34:06.441167+0000","last_peered":"2024-03-24T11:34:06.441167+0000","last_clean":"2024-03-24T11:34:06.441167+0000","last_became_active":"2024-03-24T11:30:35.353946+0000","last_became_peered":"2024-03-24T11:30:35.353946+0000","last_unstale":"2024-03-24T11:34:06.441167+0000","last_undegraded":"2024-03-24T11:34:06.441167+0000","last_fullsized":"2024-03-24T11:34:06.441167+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:30.728004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.728004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":714,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438085+0000","last_change":"2024-03-24T11:31:36.682644+0000","last_active":"2024-03-24T11:34:06.438085+0000","last_peered":"2024-03-24T11:34:06.438085+0000","last_clean":"2024-03-24T11:34:06.438085+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:34:06.438085+0000","last_undegraded":"2024-03-24T11:34:06.438085+0000","last_fullsized":"2024-03-24T11:34:06.438085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:36.682542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.682542+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":511,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.452405+0000","last_change":"2024-03-24T11:32:31.071719+0000","last_active":"2024-03-24T11:34:06.452405+0000","last_peered":"2024-03-24T11:34:06.452405+0000","last_clean":"2024-03-24T11:34:06.452405+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:34:06.452405+0000","last_undegraded":"2024-03-24T11:34:06.452405+0000","last_fullsized":"2024-03-24T11:34:06.452405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:31.071621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:31.071621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.448679+0000","last_change":"2024-03-24T11:33:30.220765+0000","last_active":"2024-03-24T11:34:06.448679+0000","last_peered":"2024-03-24T11:34:06.448679+0000","last_clean":"2024-03-24T11:34:06.448679+0000","last_became_active":"2024-03-24T11:28:43.188012+0000","last_became_peered":"2024-03-24T11:28:43.188012+0000","last_unstale":"2024-03-24T11:34:06.448679+0000","last_undegraded":"2024-03-24T11:34:06.448679+0000","last_fullsized":"2024-03-24T11:34:06.448679+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:30.220639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:30.220639+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:43.718076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"213'24","reported_seq":563,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441289+0000","last_change":"2024-03-24T11:32:43.646863+0000","last_active":"2024-03-24T11:34:06.441289+0000","last_peered":"2024-03-24T11:34:06.441289+0000","last_clean":"2024-03-24T11:34:06.441289+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:34:06.441289+0000","last_undegraded":"2024-03-24T11:34:06.441289+0000","last_fullsized":"2024-03-24T11:34:06.441289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:43.646773+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.646773+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":523,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441363+0000","last_change":"2024-03-24T11:31:37.636078+0000","last_active":"2024-03-24T11:34:06.441363+0000","last_peered":"2024-03-24T11:34:06.441363+0000","last_clean":"2024-03-24T11:34:06.441363+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:34:06.441363+0000","last_undegraded":"2024-03-24T11:34:06.441363+0000","last_fullsized":"2024-03-24T11:34:06.441363+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:37.635705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.635705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":539,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437458+0000","last_change":"2024-03-24T11:32:36.752484+0000","last_active":"2024-03-24T11:34:06.437458+0000","last_peered":"2024-03-24T11:34:06.437458+0000","last_clean":"2024-03-24T11:34:06.437458+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:34:06.437458+0000","last_undegraded":"2024-03-24T11:34:06.437458+0000","last_fullsized":"2024-03-24T11:34:06.437458+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.752406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.752406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":503,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.456825+0000","last_change":"2024-03-24T11:32:32.077273+0000","last_active":"2024-03-24T11:34:06.456825+0000","last_peered":"2024-03-24T11:34:06.456825+0000","last_clean":"2024-03-24T11:34:06.456825+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:34:06.456825+0000","last_undegraded":"2024-03-24T11:34:06.456825+0000","last_fullsized":"2024-03-24T11:34:06.456825+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.077150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.077150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"213'15","reported_seq":560,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437628+0000","last_change":"2024-03-24T11:33:28.304466+0000","last_active":"2024-03-24T11:34:06.437628+0000","last_peered":"2024-03-24T11:34:06.437628+0000","last_clean":"2024-03-24T11:34:06.437628+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:34:06.437628+0000","last_undegraded":"2024-03-24T11:34:06.437628+0000","last_fullsized":"2024-03-24T11:34:06.437628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:33:28.304078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:28.304078+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:46.020963+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.461970+0000","last_change":"2024-03-24T11:32:45.875224+0000","last_active":"2024-03-24T11:34:06.461970+0000","last_peered":"2024-03-24T11:34:06.461970+0000","last_clean":"2024-03-24T11:34:06.461970+0000","last_became_active":"2024-03-24T11:30:35.353230+0000","last_became_peered":"2024-03-24T11:30:35.353230+0000","last_unstale":"2024-03-24T11:34:06.461970+0000","last_undegraded":"2024-03-24T11:34:06.461970+0000","last_fullsized":"2024-03-24T11:34:06.461970+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.875145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.875145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:14.877519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":507,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.455994+0000","last_change":"2024-03-24T11:33:26.388166+0000","last_active":"2024-03-24T11:34:06.455994+0000","last_peered":"2024-03-24T11:34:06.455994+0000","last_clean":"2024-03-24T11:34:06.455994+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:34:06.455994+0000","last_undegraded":"2024-03-24T11:34:06.455994+0000","last_fullsized":"2024-03-24T11:34:06.455994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:26.387703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:26.387703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:44.907638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":511,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.448085+0000","last_change":"2024-03-24T11:32:24.051941+0000","last_active":"2024-03-24T11:34:06.448085+0000","last_peered":"2024-03-24T11:34:06.448085+0000","last_clean":"2024-03-24T11:34:06.448085+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:34:06.448085+0000","last_undegraded":"2024-03-24T11:34:06.448085+0000","last_fullsized":"2024-03-24T11:34:06.448085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.051777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.051777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":672,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.443583+0000","last_change":"2024-03-24T11:32:40.219356+0000","last_active":"2024-03-24T11:34:06.443583+0000","last_peered":"2024-03-24T11:34:06.443583+0000","last_clean":"2024-03-24T11:34:06.443583+0000","last_became_active":"2024-03-24T11:28:43.192919+0000","last_became_peered":"2024-03-24T11:28:43.192919+0000","last_unstale":"2024-03-24T11:34:06.443583+0000","last_undegraded":"2024-03-24T11:34:06.443583+0000","last_fullsized":"2024-03-24T11:34:06.443583+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:40.219272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.219272+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"213'19","reported_seq":542,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.457168+0000","last_change":"2024-03-24T11:33:40.546128+0000","last_active":"2024-03-24T11:34:06.457168+0000","last_peered":"2024-03-24T11:34:06.457168+0000","last_clean":"2024-03-24T11:34:06.457168+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:34:06.457168+0000","last_undegraded":"2024-03-24T11:34:06.457168+0000","last_fullsized":"2024-03-24T11:34:06.457168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:33:40.545997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:40.545997+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:59.105058+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438345+0000","last_change":"2024-03-24T11:31:40.668063+0000","last_active":"2024-03-24T11:34:06.438345+0000","last_peered":"2024-03-24T11:34:06.438345+0000","last_clean":"2024-03-24T11:34:06.438345+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:34:06.438345+0000","last_undegraded":"2024-03-24T11:34:06.438345+0000","last_fullsized":"2024-03-24T11:34:06.438345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:40.667849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:40.667849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441038+0000","last_change":"2024-03-24T11:32:23.732096+0000","last_active":"2024-03-24T11:34:06.441038+0000","last_peered":"2024-03-24T11:34:06.441038+0000","last_clean":"2024-03-24T11:34:06.441038+0000","last_became_active":"2024-03-24T11:30:35.353571+0000","last_became_peered":"2024-03-24T11:30:35.353571+0000","last_unstale":"2024-03-24T11:34:06.441038+0000","last_undegraded":"2024-03-24T11:34:06.441038+0000","last_fullsized":"2024-03-24T11:34:06.441038+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:23.731967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.731967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":503,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.456243+0000","last_change":"2024-03-24T11:30:24.313618+0000","last_active":"2024-03-24T11:34:06.456243+0000","last_peered":"2024-03-24T11:34:06.456243+0000","last_clean":"2024-03-24T11:34:06.456243+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:34:06.456243+0000","last_undegraded":"2024-03-24T11:34:06.456243+0000","last_fullsized":"2024-03-24T11:34:06.456243+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:24.313152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.313152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"213'10","reported_seq":525,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.454756+0000","last_change":"2024-03-24T11:33:33.501770+0000","last_active":"2024-03-24T11:34:06.454756+0000","last_peered":"2024-03-24T11:34:06.454756+0000","last_clean":"2024-03-24T11:34:06.454756+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:34:06.454756+0000","last_undegraded":"2024-03-24T11:34:06.454756+0000","last_fullsized":"2024-03-24T11:34:06.454756+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:33:33.501639+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.501639+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:38.233532+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449422+0000","last_change":"2024-03-24T11:32:36.013375+0000","last_active":"2024-03-24T11:34:06.449422+0000","last_peered":"2024-03-24T11:34:06.449422+0000","last_clean":"2024-03-24T11:34:06.449422+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:34:06.449422+0000","last_undegraded":"2024-03-24T11:34:06.449422+0000","last_fullsized":"2024-03-24T11:34:06.449422+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.013287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.013287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":539,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.444954+0000","last_change":"2024-03-24T11:32:38.991091+0000","last_active":"2024-03-24T11:34:06.444954+0000","last_peered":"2024-03-24T11:34:06.444954+0000","last_clean":"2024-03-24T11:34:06.444954+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:34:06.444954+0000","last_undegraded":"2024-03-24T11:34:06.444954+0000","last_fullsized":"2024-03-24T11:34:06.444954+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.991009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.991009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":143,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.443287+0000","last_change":"2024-03-24T11:32:43.213245+0000","last_active":"2024-03-24T11:34:06.443287+0000","last_peered":"2024-03-24T11:34:06.443287+0000","last_clean":"2024-03-24T11:34:06.443287+0000","last_became_active":"2024-03-24T11:30:35.353704+0000","last_became_peered":"2024-03-24T11:30:35.353704+0000","last_unstale":"2024-03-24T11:34:06.443287+0000","last_undegraded":"2024-03-24T11:34:06.443287+0000","last_fullsized":"2024-03-24T11:34:06.443287+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:43.213130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.213130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"295'5220","reported_seq":5443,"reported_epoch":295,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-24T11:34:09.772382+0000","last_change":"2024-03-24T11:33:49.745656+0000","last_active":"2024-03-24T11:34:09.772382+0000","last_peered":"2024-03-24T11:34:09.772382+0000","last_clean":"2024-03-24T11:33:47.273586+0000","last_became_active":"2024-03-24T11:33:49.339394+0000","last_became_peered":"2024-03-24T11:33:49.339394+0000","last_unstale":"2024-03-24T11:34:09.772382+0000","last_undegraded":"2024-03-24T11:34:09.772382+0000","last_fullsized":"2024-03-24T11:34:09.772382+0000","mapping_epoch":291,"log_start":"272'600","ondisk_log_start":"272'600","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187957248,"num_objects":2868,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2786,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":183552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4718592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[0,7,4],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":2868}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"213'24","reported_seq":580,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.446025+0000","last_change":"2024-03-24T11:32:30.897616+0000","last_active":"2024-03-24T11:34:06.446025+0000","last_peered":"2024-03-24T11:34:06.446025+0000","last_clean":"2024-03-24T11:34:06.446025+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:34:06.446025+0000","last_undegraded":"2024-03-24T11:34:06.446025+0000","last_fullsized":"2024-03-24T11:34:06.446025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:30.897405+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.897405+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":515,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.454650+0000","last_change":"2024-03-24T11:32:30.073961+0000","last_active":"2024-03-24T11:34:06.454650+0000","last_peered":"2024-03-24T11:34:06.454650+0000","last_clean":"2024-03-24T11:34:06.454650+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:34:06.454650+0000","last_undegraded":"2024-03-24T11:34:06.454650+0000","last_fullsized":"2024-03-24T11:34:06.454650+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:30.073873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.073873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":524,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.455064+0000","last_change":"2024-03-24T11:33:34.153970+0000","last_active":"2024-03-24T11:34:06.455064+0000","last_peered":"2024-03-24T11:34:06.455064+0000","last_clean":"2024-03-24T11:34:06.455064+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:34:06.455064+0000","last_undegraded":"2024-03-24T11:34:06.455064+0000","last_fullsized":"2024-03-24T11:34:06.455064+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:34.153751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:33:34.153751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:54.715246+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":527,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438226+0000","last_change":"2024-03-24T11:31:41.853862+0000","last_active":"2024-03-24T11:34:06.438226+0000","last_peered":"2024-03-24T11:34:06.438226+0000","last_clean":"2024-03-24T11:34:06.438226+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:34:06.438226+0000","last_undegraded":"2024-03-24T11:34:06.438226+0000","last_fullsized":"2024-03-24T11:34:06.438226+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.853751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"295'5157","reported_seq":10039,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:08.247704+0000","last_change":"2024-03-24T11:34:06.449939+0000","last_active":"2024-03-24T11:34:08.247704+0000","last_peered":"2024-03-24T11:34:08.247704+0000","last_clean":"2024-03-24T11:34:08.247704+0000","last_became_active":"2024-03-24T11:34:06.445898+0000","last_became_peered":"2024-03-24T11:34:06.445898+0000","last_unstale":"2024-03-24T11:34:08.247704+0000","last_undegraded":"2024-03-24T11:34:08.247704+0000","last_fullsized":"2024-03-24T11:34:08.247704+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:11.971653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185729024,"num_objects":2834,"num_object_clones":0,"num_object_copies":8502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":181376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2399,"num_bytes_recovered":157220864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"213'20","reported_seq":584,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.452603+0000","last_change":"2024-03-24T11:33:41.275622+0000","last_active":"2024-03-24T11:34:06.452603+0000","last_peered":"2024-03-24T11:34:06.452603+0000","last_clean":"2024-03-24T11:34:06.452603+0000","last_became_active":"2024-03-24T11:33:33.699957+0000","last_became_peered":"2024-03-24T11:33:33.699957+0000","last_unstale":"2024-03-24T11:34:06.452603+0000","last_undegraded":"2024-03-24T11:34:06.452603+0000","last_fullsized":"2024-03-24T11:34:06.452603+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:41.275381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:41.275381+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:47.265254+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":548,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437080+0000","last_change":"2024-03-24T11:33:31.339856+0000","last_active":"2024-03-24T11:34:06.437080+0000","last_peered":"2024-03-24T11:34:06.437080+0000","last_clean":"2024-03-24T11:34:06.437080+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:34:06.437080+0000","last_undegraded":"2024-03-24T11:34:06.437080+0000","last_fullsized":"2024-03-24T11:34:06.437080+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:33:31.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:31.339716+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:34.075394+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":738,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.445938+0000","last_change":"2024-03-24T11:32:41.963960+0000","last_active":"2024-03-24T11:34:06.445938+0000","last_peered":"2024-03-24T11:34:06.445938+0000","last_clean":"2024-03-24T11:34:06.445938+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:34:06.445938+0000","last_undegraded":"2024-03-24T11:34:06.445938+0000","last_fullsized":"2024-03-24T11:34:06.445938+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:32:41.963825+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.963825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":182,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437844+0000","last_change":"2024-03-24T11:33:27.437600+0000","last_active":"2024-03-24T11:34:06.437844+0000","last_peered":"2024-03-24T11:34:06.437844+0000","last_clean":"2024-03-24T11:34:06.437844+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:34:06.437844+0000","last_undegraded":"2024-03-24T11:34:06.437844+0000","last_fullsized":"2024-03-24T11:34:06.437844+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:33:27.437477+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:33:27.437477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:31.901193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.13","version":"295'5364","reported_seq":5443,"reported_epoch":295,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-24T11:34:08.575084+0000","last_change":"2024-03-24T11:33:49.740860+0000","last_active":"2024-03-24T11:34:08.575084+0000","last_peered":"2024-03-24T11:34:08.575084+0000","last_clean":"2024-03-24T11:33:47.273303+0000","last_became_active":"2024-03-24T11:33:49.728951+0000","last_became_peered":"2024-03-24T11:33:49.728951+0000","last_unstale":"2024-03-24T11:34:08.575084+0000","last_undegraded":"2024-03-24T11:33:49.595750+0000","last_fullsized":"2024-03-24T11:33:49.282397+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192741376,"num_objects":2941,"num_object_clones":0,"num_object_copies":8823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2941,"num_write_kb":188224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[2,1],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"1,2","objects":437},{"shards":"2,3,5","objects":2504}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"213'18","reported_seq":535,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.462300+0000","last_change":"2024-03-24T11:31:11.260142+0000","last_active":"2024-03-24T11:34:06.462300+0000","last_peered":"2024-03-24T11:34:06.462300+0000","last_clean":"2024-03-24T11:34:06.462300+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:34:06.462300+0000","last_undegraded":"2024-03-24T11:34:06.462300+0000","last_fullsized":"2024-03-24T11:34:06.462300+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.259991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.259991+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":523,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441372+0000","last_change":"2024-03-24T11:33:19.752899+0000","last_active":"2024-03-24T11:34:06.441372+0000","last_peered":"2024-03-24T11:34:06.441372+0000","last_clean":"2024-03-24T11:34:06.441372+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:34:06.441372+0000","last_undegraded":"2024-03-24T11:34:06.441372+0000","last_fullsized":"2024-03-24T11:34:06.441372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.752717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.752717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:25.705201+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":523,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441649+0000","last_change":"2024-03-24T11:32:38.742586+0000","last_active":"2024-03-24T11:34:06.441649+0000","last_peered":"2024-03-24T11:34:06.441649+0000","last_clean":"2024-03-24T11:34:06.441649+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:34:06.441649+0000","last_undegraded":"2024-03-24T11:34:06.441649+0000","last_fullsized":"2024-03-24T11:34:06.441649+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.742478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.742478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":499,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.458055+0000","last_change":"2024-03-24T11:33:44.241415+0000","last_active":"2024-03-24T11:34:06.458055+0000","last_peered":"2024-03-24T11:34:06.458055+0000","last_clean":"2024-03-24T11:34:06.458055+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:34:06.458055+0000","last_undegraded":"2024-03-24T11:34:06.458055+0000","last_fullsized":"2024-03-24T11:34:06.458055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:08.308860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:08.308860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"295'5279","reported_seq":5723,"reported_epoch":295,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-24T11:34:09.805292+0000","last_change":"2024-03-24T11:34:05.290712+0000","last_active":"2024-03-24T11:34:09.805292+0000","last_peered":"2024-03-24T11:34:09.805292+0000","last_clean":"2024-03-24T11:33:47.181830+0000","last_became_active":"2024-03-24T11:33:49.713894+0000","last_became_peered":"2024-03-24T11:33:49.713894+0000","last_unstale":"2024-03-24T11:34:09.805292+0000","last_undegraded":"2024-03-24T11:34:09.805292+0000","last_fullsized":"2024-03-24T11:33:49.286259+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190545920,"num_objects":2908,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7215,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":186080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[0,2],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":2405},{"shards":"0,2","objects":503}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"213'27","reported_seq":586,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.443720+0000","last_change":"2024-03-24T11:33:40.976361+0000","last_active":"2024-03-24T11:34:06.443720+0000","last_peered":"2024-03-24T11:34:06.443720+0000","last_clean":"2024-03-24T11:34:06.443720+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:34:06.443720+0000","last_undegraded":"2024-03-24T11:34:06.443720+0000","last_fullsized":"2024-03-24T11:34:06.443720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:33:40.976211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:40.976211+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:50.179213+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449836+0000","last_change":"2024-03-24T11:33:22.182976+0000","last_active":"2024-03-24T11:34:06.449836+0000","last_peered":"2024-03-24T11:34:06.449836+0000","last_clean":"2024-03-24T11:34:06.449836+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:34:06.449836+0000","last_undegraded":"2024-03-24T11:34:06.449836+0000","last_fullsized":"2024-03-24T11:34:06.449836+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:22.182859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.182859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:32.867301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.438100+0000","last_change":"2024-03-24T11:32:42.808822+0000","last_active":"2024-03-24T11:34:06.438100+0000","last_peered":"2024-03-24T11:34:06.438100+0000","last_clean":"2024-03-24T11:34:06.438100+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:34:06.438100+0000","last_undegraded":"2024-03-24T11:34:06.438100+0000","last_fullsized":"2024-03-24T11:34:06.438100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.808738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.808738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":520,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441689+0000","last_change":"2024-03-24T11:33:42.035003+0000","last_active":"2024-03-24T11:34:06.441689+0000","last_peered":"2024-03-24T11:34:06.441689+0000","last_clean":"2024-03-24T11:34:06.441689+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:34:06.441689+0000","last_undegraded":"2024-03-24T11:34:06.441689+0000","last_fullsized":"2024-03-24T11:34:06.441689+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.034893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.034893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:10.513494+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"295'5296","reported_seq":7200,"reported_epoch":295,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-24T11:34:09.620116+0000","last_change":"2024-03-24T11:33:49.650122+0000","last_active":"2024-03-24T11:34:09.620116+0000","last_peered":"2024-03-24T11:34:09.620116+0000","last_clean":"2024-03-24T11:33:47.258594+0000","last_became_active":"2024-03-24T11:33:49.593808+0000","last_became_peered":"2024-03-24T11:33:49.593808+0000","last_unstale":"2024-03-24T11:34:09.620116+0000","last_undegraded":"2024-03-24T11:34:09.620116+0000","last_fullsized":"2024-03-24T11:33:49.281397+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190545920,"num_objects":2908,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3678,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":186080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":39911424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,1],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":1839},{"shards":"1,3","objects":1069}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"213'7","reported_seq":532,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.440948+0000","last_change":"2024-03-24T11:31:27.652238+0000","last_active":"2024-03-24T11:34:06.440948+0000","last_peered":"2024-03-24T11:34:06.440948+0000","last_clean":"2024-03-24T11:34:06.440948+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:34:06.440948+0000","last_undegraded":"2024-03-24T11:34:06.440948+0000","last_fullsized":"2024-03-24T11:34:06.440948+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:31:27.652037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.652037+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437970+0000","last_change":"2024-03-24T11:33:19.374929+0000","last_active":"2024-03-24T11:34:06.437970+0000","last_peered":"2024-03-24T11:34:06.437970+0000","last_clean":"2024-03-24T11:34:06.437970+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:34:06.437970+0000","last_undegraded":"2024-03-24T11:34:06.437970+0000","last_fullsized":"2024-03-24T11:34:06.437970+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.374805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.374805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:37.994295+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":135,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.442100+0000","last_change":"2024-03-24T11:31:41.715026+0000","last_active":"2024-03-24T11:34:06.442100+0000","last_peered":"2024-03-24T11:34:06.442100+0000","last_clean":"2024-03-24T11:34:06.442100+0000","last_became_active":"2024-03-24T11:30:35.359239+0000","last_became_peered":"2024-03-24T11:30:35.359239+0000","last_unstale":"2024-03-24T11:34:06.442100+0000","last_undegraded":"2024-03-24T11:34:06.442100+0000","last_fullsized":"2024-03-24T11:34:06.442100+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437609+0000","last_change":"2024-03-24T11:33:28.608037+0000","last_active":"2024-03-24T11:34:06.437609+0000","last_peered":"2024-03-24T11:34:06.437609+0000","last_clean":"2024-03-24T11:34:06.437609+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:34:06.437609+0000","last_undegraded":"2024-03-24T11:34:06.437609+0000","last_fullsized":"2024-03-24T11:34:06.437609+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:33:28.607919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:54.542694+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"295'5327","reported_seq":5402,"reported_epoch":295,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-24T11:34:09.685707+0000","last_change":"2024-03-24T11:33:49.724865+0000","last_active":"2024-03-24T11:34:09.685707+0000","last_peered":"2024-03-24T11:34:09.685707+0000","last_clean":"2024-03-24T11:33:47.274322+0000","last_became_active":"2024-03-24T11:33:49.671548+0000","last_became_peered":"2024-03-24T11:33:49.671548+0000","last_unstale":"2024-03-24T11:34:09.685707+0000","last_undegraded":"2024-03-24T11:33:49.654818+0000","last_fullsized":"2024-03-24T11:33:49.315960+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5327,"log_dups_size":0,"ondisk_log_size":5327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192741376,"num_objects":2941,"num_object_clones":0,"num_object_copies":8823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2448,"num_objects_misplaced":2448,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2941,"num_write_kb":188224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[0,4],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":2448},{"shards":"0,4","objects":493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"213'15","reported_seq":544,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441109+0000","last_change":"2024-03-24T11:33:24.846253+0000","last_active":"2024-03-24T11:34:06.441109+0000","last_peered":"2024-03-24T11:34:06.441109+0000","last_clean":"2024-03-24T11:34:06.441109+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:34:06.441109+0000","last_undegraded":"2024-03-24T11:34:06.441109+0000","last_fullsized":"2024-03-24T11:34:06.441109+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:33:24.845898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.845898+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:46.084580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437993+0000","last_change":"2024-03-24T11:31:41.687584+0000","last_active":"2024-03-24T11:34:06.437993+0000","last_peered":"2024-03-24T11:34:06.437993+0000","last_clean":"2024-03-24T11:34:06.437993+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:34:06.437993+0000","last_undegraded":"2024-03-24T11:34:06.437993+0000","last_fullsized":"2024-03-24T11:34:06.437993+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.687474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.687474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.437729+0000","last_change":"2024-03-24T11:33:38.555137+0000","last_active":"2024-03-24T11:34:06.437729+0000","last_peered":"2024-03-24T11:34:06.437729+0000","last_clean":"2024-03-24T11:34:06.437729+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:34:06.437729+0000","last_undegraded":"2024-03-24T11:34:06.437729+0000","last_fullsized":"2024-03-24T11:34:06.437729+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:38.554887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.554887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:42.855086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449674+0000","last_change":"2024-03-24T11:33:24.183098+0000","last_active":"2024-03-24T11:34:06.449674+0000","last_peered":"2024-03-24T11:34:06.449674+0000","last_clean":"2024-03-24T11:34:06.449674+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:34:06.449674+0000","last_undegraded":"2024-03-24T11:34:06.449674+0000","last_fullsized":"2024-03-24T11:34:06.449674+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:24.182966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.182966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:34:34.665679+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"295'5224","reported_seq":7252,"reported_epoch":295,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-24T11:34:09.613431+0000","last_change":"2024-03-24T11:33:49.446412+0000","last_active":"2024-03-24T11:34:09.613431+0000","last_peered":"2024-03-24T11:34:09.613431+0000","last_clean":"2024-03-24T11:33:47.266438+0000","last_became_active":"2024-03-24T11:33:49.423359+0000","last_became_peered":"2024-03-24T11:33:49.423359+0000","last_unstale":"2024-03-24T11:34:09.613431+0000","last_undegraded":"2024-03-24T11:34:09.613431+0000","last_fullsized":"2024-03-24T11:33:49.280752+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187990016,"num_objects":2869,"num_object_clones":0,"num_object_copies":8607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1412,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":183584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":63963136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[3,6],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1412},{"shards":"3,6","objects":1457}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"213'27","reported_seq":546,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.457257+0000","last_change":"2024-03-24T11:31:41.107651+0000","last_active":"2024-03-24T11:34:06.457257+0000","last_peered":"2024-03-24T11:34:06.457257+0000","last_clean":"2024-03-24T11:34:06.457257+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:34:06.457257+0000","last_undegraded":"2024-03-24T11:34:06.457257+0000","last_fullsized":"2024-03-24T11:34:06.457257+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:31:41.107512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.107512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":508,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.456443+0000","last_change":"2024-03-24T11:30:23.137546+0000","last_active":"2024-03-24T11:34:06.456443+0000","last_peered":"2024-03-24T11:34:06.456443+0000","last_clean":"2024-03-24T11:34:06.456443+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:34:06.456443+0000","last_undegraded":"2024-03-24T11:34:06.456443+0000","last_fullsized":"2024-03-24T11:34:06.456443+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:23.137393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:30:23.137393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":515,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441397+0000","last_change":"2024-03-24T11:32:24.692923+0000","last_active":"2024-03-24T11:34:06.441397+0000","last_peered":"2024-03-24T11:34:06.441397+0000","last_clean":"2024-03-24T11:34:06.441397+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:34:06.441397+0000","last_undegraded":"2024-03-24T11:34:06.441397+0000","last_fullsized":"2024-03-24T11:34:06.441397+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.692844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.692844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":495,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.456077+0000","last_change":"2024-03-24T11:31:14.327893+0000","last_active":"2024-03-24T11:34:06.456077+0000","last_peered":"2024-03-24T11:34:06.456077+0000","last_clean":"2024-03-24T11:34:06.456077+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:34:06.456077+0000","last_undegraded":"2024-03-24T11:34:06.456077+0000","last_fullsized":"2024-03-24T11:34:06.456077+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.327777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.327777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"295'5330","reported_seq":6955,"reported_epoch":295,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-24T11:34:09.881793+0000","last_change":"2024-03-24T11:33:49.721372+0000","last_active":"2024-03-24T11:34:09.881793+0000","last_peered":"2024-03-24T11:34:09.881793+0000","last_clean":"2024-03-24T11:33:47.222522+0000","last_became_active":"2024-03-24T11:33:49.693719+0000","last_became_peered":"2024-03-24T11:33:49.693719+0000","last_unstale":"2024-03-24T11:34:09.881793+0000","last_undegraded":"2024-03-24T11:34:09.881793+0000","last_fullsized":"2024-03-24T11:33:49.289445+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":288,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192806912,"num_objects":2942,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3784,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2942,"num_write_kb":188288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":33882112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[0,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":1892},{"shards":"0,5","objects":1050}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"213'15","reported_seq":568,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441364+0000","last_change":"2024-03-24T11:33:38.953403+0000","last_active":"2024-03-24T11:34:06.441364+0000","last_peered":"2024-03-24T11:34:06.441364+0000","last_clean":"2024-03-24T11:34:06.441364+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:34:06.441364+0000","last_undegraded":"2024-03-24T11:34:06.441364+0000","last_fullsized":"2024-03-24T11:34:06.441364+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:33:38.953051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.953051+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:00.825426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":524,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.449217+0000","last_change":"2024-03-24T11:32:35.031834+0000","last_active":"2024-03-24T11:34:06.449217+0000","last_peered":"2024-03-24T11:34:06.449217+0000","last_clean":"2024-03-24T11:34:06.449217+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:34:06.449217+0000","last_undegraded":"2024-03-24T11:34:06.449217+0000","last_fullsized":"2024-03-24T11:34:06.449217+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:35.031735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:35.031735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":531,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.436551+0000","last_change":"2024-03-24T11:32:28.712422+0000","last_active":"2024-03-24T11:34:06.436551+0000","last_peered":"2024-03-24T11:34:06.436551+0000","last_clean":"2024-03-24T11:34:06.436551+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:34:06.436551+0000","last_undegraded":"2024-03-24T11:34:06.436551+0000","last_fullsized":"2024-03-24T11:34:06.436551+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.712294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.712294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":519,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-24T11:34:06.441504+0000","last_change":"2024-03-24T11:33:36.020005+0000","last_active":"2024-03-24T11:34:06.441504+0000","last_peered":"2024-03-24T11:34:06.441504+0000","last_clean":"2024-03-24T11:34:06.441504+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:34:06.441504+0000","last_undegraded":"2024-03-24T11:34:06.441504+0000","last_fullsized":"2024-03-24T11:34:06.441504+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:36.019890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:36.019890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:35:04.756527+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":6055657472,"num_objects":92402,"num_object_clones":0,"num_object_copies":277206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12278,"num_objects_misplaced":27612,"num_objects_unfound":0,"num_objects_dirty":92402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92402,"num_write_kb":5913728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5205,"num_bytes_recovered":341114880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18593677312,"data_stored":18593677312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131486,"ondisk_log_size":131486,"up":78,"acting":70,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070651,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2166820,"kb_used_data":1621972,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91578332,"statfs":{"total":95995035648,"available":93776211968,"internally_reserved":0,"allocated":1660899328,"data_stored":1660458033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":557906379},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364992,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3043920,"kb_used_data":2643464,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":90701232,"statfs":{"total":95995035648,"available":92878061568,"internally_reserved":0,"allocated":2706907136,"data_stored":2706478350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":410057158},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1720764,"kb_used_data":1157872,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92024388,"statfs":{"total":95995035648,"available":94232973312,"internally_reserved":0,"allocated":1185660928,"data_stored":1185192019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4221028,"kb_used_data":3644252,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":89524124,"statfs":{"total":95995035648,"available":91672702976,"internally_reserved":0,"allocated":3731714048,"data_stored":3731264316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411541,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2670048,"kb_used_data":2255700,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91075104,"statfs":{"total":95995035648,"available":93260906496,"internally_reserved":0,"allocated":2309836800,"data_stored":2309395565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149902,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2444792,"kb_used_data":1886064,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91300360,"statfs":{"total":95995035648,"available":93491568640,"internally_reserved":0,"allocated":1931329536,"data_stored":1930860363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":572127687},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986378,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3381996,"kb_used_data":2989856,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":90363156,"statfs":{"total":95995035648,"available":92531871744,"internally_reserved":0,"allocated":3061612544,"data_stored":3061159389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659333,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2429848,"kb_used_data":1871120,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91315304,"statfs":{"total":95995035648,"available":93506871296,"internally_reserved":0,"allocated":1916026880,"data_stored":1915590007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3754885120,"data_stored":3754885120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1196228608,"data_stored":1196228608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2324758528,"data_stored":2324758528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1941241856,"data_stored":1941241856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3069575168,"data_stored":3069575168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1917517824,"data_stored":1917517824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2718105600,"data_stored":2718105600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1671364608,"data_stored":1671364608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:34:36.734 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-03-24T11:34:36.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-24T11:34:36.793 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:36.856 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:36.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:36.898 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:36.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:36 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:34:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:36 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:34:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:36 smithi181 ceph-mon[33998]: osdmap e300: 8 total, 8 up, 8 in 2024-03-24T11:34:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3332951211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:34:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:36 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:34:36.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:36.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:36.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:37.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:37.114 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:37.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:37.216 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:37.272 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:37.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:37.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:37.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:37.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:37.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:37.553 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:37.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:37.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:37.684 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:37.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:37.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[29608]: pgmap v672: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 671 B/s rd, 32 MiB/s wr, 510 op/s; 12278/330942 objects degraded (3.710%); 20500/330942 objects misplaced (6.194%); 10 MiB/s, 164 objects/s recovering 2024-03-24T11:34:37.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:34:37.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:34:37.712 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[29608]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T11:34:37.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[37907]: pgmap v672: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 671 B/s rd, 32 MiB/s wr, 510 op/s; 12278/330942 objects degraded (3.710%); 20500/330942 objects misplaced (6.194%); 10 MiB/s, 164 objects/s recovering 2024-03-24T11:34:37.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:34:37.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:34:37.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:37 smithi179 ceph-mon[37907]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T11:34:37.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:37.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:37.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:37 smithi181 ceph-mon[33998]: pgmap v672: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 671 B/s rd, 32 MiB/s wr, 510 op/s; 12278/330942 objects degraded (3.710%); 20500/330942 objects misplaced (6.194%); 10 MiB/s, 164 objects/s recovering 2024-03-24T11:34:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:37 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:34:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:37 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:34:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:37 smithi181 ceph-mon[33998]: osdmap e301: 8 total, 8 up, 8 in 2024-03-24T11:34:37.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:37.979 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:37.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:38.081 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:38.111 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:38.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:38.213 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:38.259 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:38.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:38.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:38.404 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:38.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:38.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:38.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:38.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:38.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:38.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:38.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:38.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:38.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:38.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:38.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:38.939 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:38.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:39.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:39.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:39.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:39.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:39.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:39.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:39.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:39.341 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:39.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:39.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:39.487 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:39.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:39.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:39.588 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:34:39.619 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:39.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:39.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:39 smithi179 ceph-mon[29608]: pgmap v674: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 35 MiB/s wr, 911 op/s; 11834/325596 objects degraded (3.635%); 19184/325596 objects misplaced (5.892%); 18 MiB/s, 287 objects/s recovering 2024-03-24T11:34:39.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:39 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 11834/325596 objects degraded (3.635%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:39.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:39 smithi179 ceph-mon[37907]: pgmap v674: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 35 MiB/s wr, 911 op/s; 11834/325596 objects degraded (3.635%); 19184/325596 objects misplaced (5.892%); 18 MiB/s, 287 objects/s recovering 2024-03-24T11:34:39.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:39 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 11834/325596 objects degraded (3.635%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:39.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:39.760 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:39.865 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:39.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:39.875 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:34:39.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:39 smithi181 ceph-mon[33998]: pgmap v674: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 35 MiB/s wr, 911 op/s; 11834/325596 objects degraded (3.635%); 19184/325596 objects misplaced (5.892%); 18 MiB/s, 287 objects/s recovering 2024-03-24T11:34:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:39 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 11834/325596 objects degraded (3.635%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:39.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:40.043 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:40.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:40.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:40.174 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:40.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:40.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:40.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:40.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:40.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:40.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:40.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:40.537 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:40.593 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:40.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:40.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:40.742 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:40.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:40.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:40.876 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:40.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:40.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:41.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:41.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:41.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:41.156 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:41.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:41.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:41.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:41.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:41.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:41.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:41.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:41.564 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:41.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:41.666 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:41.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:34:41.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:34:41.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[29608]: pgmap v675: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1006 B/s rd, 4.5 MiB/s wr, 672 op/s; 11834/321081 objects degraded (3.686%); 18628/321081 objects misplaced (5.802%); 13 MiB/s, 211 objects/s recovering 2024-03-24T11:34:41.686 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[29608]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T11:34:41.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:34:41.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:34:41.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[37907]: pgmap v675: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1006 B/s rd, 4.5 MiB/s wr, 672 op/s; 11834/321081 objects degraded (3.686%); 18628/321081 objects misplaced (5.802%); 13 MiB/s, 211 objects/s recovering 2024-03-24T11:34:41.687 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:41 smithi179 ceph-mon[37907]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T11:34:41.707 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:41.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:41.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:41.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:41 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:34:41.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:41 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:34:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:41 smithi181 ceph-mon[33998]: pgmap v675: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1006 B/s rd, 4.5 MiB/s wr, 672 op/s; 11834/321081 objects degraded (3.686%); 18628/321081 objects misplaced (5.802%); 13 MiB/s, 211 objects/s recovering 2024-03-24T11:34:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:41 smithi181 ceph-mon[33998]: osdmap e302: 8 total, 8 up, 8 in 2024-03-24T11:34:41.920 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:41.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:42.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:42.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:42.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:42.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:42.175 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:42.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:42.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:42.320 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:42.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:42.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:42.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:42.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:42.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:42.600 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:42.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:42.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:42.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:42.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:42.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:42.883 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:42.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:42.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:43.023 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:43.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:43.154 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:43.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:43.288 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:43.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:43.427 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:43.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:43.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:43.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:43.684 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:43 smithi179 ceph-mon[29608]: pgmap v677: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 887 B/s rd, 3.1 MiB/s wr, 576 op/s; 11834/321081 objects degraded (3.686%); 18628/321081 objects misplaced (5.802%); 9.0 MiB/s, 144 objects/s recovering 2024-03-24T11:34:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:43 smithi179 ceph-mon[37907]: pgmap v677: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 887 B/s rd, 3.1 MiB/s wr, 576 op/s; 11834/321081 objects degraded (3.686%); 18628/321081 objects misplaced (5.802%); 9.0 MiB/s, 144 objects/s recovering 2024-03-24T11:34:43.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:43.816 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:43 smithi181 ceph-mon[33998]: pgmap v677: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 887 B/s rd, 3.1 MiB/s wr, 576 op/s; 11834/321081 objects degraded (3.686%); 18628/321081 objects misplaced (5.802%); 9.0 MiB/s, 144 objects/s recovering 2024-03-24T11:34:43.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:43.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:43.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:44.050 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:44.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:44.094 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:44.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:44.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:44.219 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:44.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:44.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:44.353 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:44.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:44.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:34:43 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:34:44.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:44.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:44.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:44.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:44.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:44.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:44.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:44.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:44.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:44.793 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:34:44.794 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:34:44.794 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[29608]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T11:34:44.794 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:44.794 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 11834/321081 objects degraded (3.686%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:44.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:34:44.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:34:44.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[37907]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T11:34:44.795 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:44.795 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:44 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 11834/321081 objects degraded (3.686%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:44.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:44.901 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:44 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:34:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:44 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:34:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:44 smithi181 ceph-mon[33998]: osdmap e303: 8 total, 8 up, 8 in 2024-03-24T11:34:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:44 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:44 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 11834/321081 objects degraded (3.686%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:44.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:45.004 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:45.047 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:45.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:45.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:45.178 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:45.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:45.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:45.308 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:45.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:45.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:45.444 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:45.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:45.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:34:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:34:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:34:45.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:45.592 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:45.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:45.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:45.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:45.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:45.826 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:34:45.826 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:34:45.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:45 smithi179 ceph-mon[29608]: pgmap v679: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.7 MiB/s wr, 1.02k op/s; 11308/308805 objects degraded (3.662%); 17097/308805 objects misplaced (5.537%); 15 MiB/s, 232 objects/s recovering 2024-03-24T11:34:45.827 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:45 smithi179 ceph-mon[37907]: pgmap v679: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.7 MiB/s wr, 1.02k op/s; 11308/308805 objects degraded (3.662%); 17097/308805 objects misplaced (5.537%); 15 MiB/s, 232 objects/s recovering 2024-03-24T11:34:45.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:45.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:45 smithi181 ceph-mon[33998]: pgmap v679: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 2.7 MiB/s wr, 1.02k op/s; 11308/308805 objects degraded (3.662%); 17097/308805 objects misplaced (5.537%); 15 MiB/s, 232 objects/s recovering 2024-03-24T11:34:45.910 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:45.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:46.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:46.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:46.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.193 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:46.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:46.294 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:46.322 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:46.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:46.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:46.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:46.578 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:46.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:46.712 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:46.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:46.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:46.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:46.968 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:46.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:47.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:47.117 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:47.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:47.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:47.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:47.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:47.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:47.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:47.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:47.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:47.591 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:47.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:47.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:47.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:47.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:47.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:47.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:47.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:47 smithi181 ceph-mon[33998]: pgmap v680: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 767 B/s rd, 488 KiB/s wr, 733 op/s; 11308/308379 objects degraded (3.667%); 16457/308379 objects misplaced (5.337%); 11 MiB/s, 179 objects/s recovering 2024-03-24T11:34:47.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:47 smithi179 ceph-mon[29608]: pgmap v680: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 767 B/s rd, 488 KiB/s wr, 733 op/s; 11308/308379 objects degraded (3.667%); 16457/308379 objects misplaced (5.337%); 11 MiB/s, 179 objects/s recovering 2024-03-24T11:34:47.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:47 smithi179 ceph-mon[37907]: pgmap v680: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 767 B/s rd, 488 KiB/s wr, 733 op/s; 11308/308379 objects degraded (3.667%); 16457/308379 objects misplaced (5.337%); 11 MiB/s, 179 objects/s recovering 2024-03-24T11:34:47.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:47.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:47.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:48.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:48.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:48.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:48.225 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:48.261 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:48.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:48.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:48.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:48.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:48.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:48.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:48.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:48.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:48.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:48.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:48.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:48.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:48.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:48.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:48.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:48 smithi181 ceph-mon[33998]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T11:34:48.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:48 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:48.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:48.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:48.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:48 smithi179 ceph-mon[29608]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T11:34:48.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:48 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:48.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:48 smithi179 ceph-mon[37907]: osdmap e304: 8 total, 8 up, 8 in 2024-03-24T11:34:48.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:48 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:34:49.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:49.034 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:49.136 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:49.181 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:49.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:49.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:49.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:49.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:49.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:49.584 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:49.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:49.719 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:49.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:49.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:49 smithi181 ceph-mon[33998]: pgmap v682: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 832 op/s; 8724/301368 objects degraded (2.895%); 16897/301368 objects misplaced (5.607%); 16 MiB/s, 257 objects/s recovering 2024-03-24T11:34:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:49 smithi181 ceph-mon[33998]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T11:34:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:49 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 8724/301368 objects degraded (2.895%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:49.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:49 smithi179 ceph-mon[29608]: pgmap v682: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 832 op/s; 8724/301368 objects degraded (2.895%); 16897/301368 objects misplaced (5.607%); 16 MiB/s, 257 objects/s recovering 2024-03-24T11:34:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:49 smithi179 ceph-mon[29608]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T11:34:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:49 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 8724/301368 objects degraded (2.895%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:49 smithi179 ceph-mon[37907]: pgmap v682: 178 pgs: 4 active+recovering+undersized+remapped, 170 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 832 op/s; 8724/301368 objects degraded (2.895%); 16897/301368 objects misplaced (5.607%); 16 MiB/s, 257 objects/s recovering 2024-03-24T11:34:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:49 smithi179 ceph-mon[37907]: osdmap e305: 8 total, 8 up, 8 in 2024-03-24T11:34:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:49 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 8724/301368 objects degraded (2.895%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T11:34:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:49.997 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:49.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:50.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:50.146 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:50.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:50.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:50.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:50.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:50.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:50.428 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:50.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:50.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:50.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:50.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:50.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:50.686 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:50.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:50.788 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:50.831 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:50.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:50.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:50.976 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:50.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:51.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:51.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:51.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:51.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:51.207 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:51.253 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:51.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:51.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:51.378 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:51.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:51.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:51.531 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:51.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:51.634 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:51.669 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:51.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:51.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:51.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:51.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:51 smithi181 ceph-mon[33998]: pgmap v684: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 612 op/s; 8724/296943 objects degraded (2.938%); 16457/296943 objects misplaced (5.542%); 13 MiB/s, 206 objects/s recovering 2024-03-24T11:34:51.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:51.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:51.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:51.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:51 smithi179 ceph-mon[29608]: pgmap v684: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 612 op/s; 8724/296943 objects degraded (2.938%); 16457/296943 objects misplaced (5.542%); 13 MiB/s, 206 objects/s recovering 2024-03-24T11:34:51.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:51 smithi179 ceph-mon[37907]: pgmap v684: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 612 op/s; 8724/296943 objects degraded (2.938%); 16457/296943 objects misplaced (5.542%); 13 MiB/s, 206 objects/s recovering 2024-03-24T11:34:52.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:52.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:52.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:52.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:52.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:52.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:52.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:52.338 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:52.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:52.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:52.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:52.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:52.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:52.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:52.620 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:52.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:52.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:52.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:52.750 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:52.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:52.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:52.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:52.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:52.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:53.027 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:53.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:53.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:53.162 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:53.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:53.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:53.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:53.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:53.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:53.425 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:53.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:53.527 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:53.569 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:53.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:53.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:53.798 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:53.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:53.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:53.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:53 smithi181 ceph-mon[33998]: pgmap v685: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 494 op/s; 8724/296943 objects degraded (2.938%); 16457/296943 objects misplaced (5.542%); 11 MiB/s, 177 objects/s recovering 2024-03-24T11:34:53.943 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:53.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:53.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:53.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:53 smithi179 ceph-mon[29608]: pgmap v685: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 494 op/s; 8724/296943 objects degraded (2.938%); 16457/296943 objects misplaced (5.542%); 11 MiB/s, 177 objects/s recovering 2024-03-24T11:34:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:53 smithi179 ceph-mon[37907]: pgmap v685: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 494 op/s; 8724/296943 objects degraded (2.938%); 16457/296943 objects misplaced (5.542%); 11 MiB/s, 177 objects/s recovering 2024-03-24T11:34:54.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:54.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:54.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:54.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:54.202 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:54.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:54.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:54.333 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:54.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:54.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:34:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:34:54.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:54.459 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:54.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:54.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:54.609 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:54.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:54.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:54.754 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:54.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:54.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:54.886 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:54.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:54.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:54 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:34:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:54 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:34:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:54 smithi181 ceph-mon[33998]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T11:34:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:54 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 8724/296943 objects degraded (2.938%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T11:34:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:54 smithi181 ceph-mon[33998]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T11:34:54.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:34:54.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:34:54.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[29608]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T11:34:54.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 8724/296943 objects degraded (2.938%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T11:34:54.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[29608]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T11:34:54.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:34:54.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:34:54.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[37907]: osdmap e306: 8 total, 8 up, 8 in 2024-03-24T11:34:54.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 8724/296943 objects degraded (2.938%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T11:34:54.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:54 smithi179 ceph-mon[37907]: osdmap e307: 8 total, 8 up, 8 in 2024-03-24T11:34:54.988 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:55.032 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:55.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:55.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:55.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:34:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:34:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:34:55.258 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:55.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:55.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:55.404 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:55.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:55.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:55.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:55.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:55.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:55.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:55.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:55.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:55.801 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:55.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:55 smithi181 ceph-mon[33998]: pgmap v688: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 853 op/s; 8304/286017 objects degraded (2.903%); 14423/286017 objects misplaced (5.043%); 13 MiB/s, 212 objects/s recovering 2024-03-24T11:34:55.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:55.928 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:55.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:55.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:55.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:55 smithi179 ceph-mon[29608]: pgmap v688: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 853 op/s; 8304/286017 objects degraded (2.903%); 14423/286017 objects misplaced (5.043%); 13 MiB/s, 212 objects/s recovering 2024-03-24T11:34:55.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:34:55.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:55 smithi179 ceph-mon[37907]: pgmap v688: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 853 op/s; 8304/286017 objects degraded (2.903%); 14423/286017 objects misplaced (5.043%); 13 MiB/s, 212 objects/s recovering 2024-03-24T11:34:56.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:56.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:56.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:56.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:56.299 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:56.323 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:56.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:56.460 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:56.562 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:56.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:56.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:56.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:56.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:56.869 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:56.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:56.995 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:56.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:57.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:57.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:57.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:57.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:57.261 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:57.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:57.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:34:57.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:57.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:34:57.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:57.533 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:57.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:57.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:57.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:57 smithi181 ceph-mon[33998]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T11:34:57.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:57 smithi181 ceph-mon[33998]: pgmap v690: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 629 op/s; 8304/285612 objects degraded (2.907%); 14423/285612 objects misplaced (5.050%); 11 MiB/s, 175 objects/s recovering 2024-03-24T11:34:57.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:57.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:57 smithi179 ceph-mon[29608]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T11:34:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:57 smithi179 ceph-mon[29608]: pgmap v690: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 629 op/s; 8304/285612 objects degraded (2.907%); 14423/285612 objects misplaced (5.050%); 11 MiB/s, 175 objects/s recovering 2024-03-24T11:34:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:57 smithi179 ceph-mon[37907]: osdmap e308: 8 total, 8 up, 8 in 2024-03-24T11:34:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:57 smithi179 ceph-mon[37907]: pgmap v690: 178 pgs: 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 629 op/s; 8304/285612 objects degraded (2.907%); 14423/285612 objects misplaced (5.050%); 11 MiB/s, 175 objects/s recovering 2024-03-24T11:34:57.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:57.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:57.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:57.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:57.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:57.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:58.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:58.057 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:58.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:58.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:58.183 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:58.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:58.285 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:58.326 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:58.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:58.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:58.456 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:58.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:58.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:58.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:58.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:34:58.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:58 smithi181 ceph-mon[33998]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T11:34:58.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:58.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:58 smithi179 ceph-mon[29608]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T11:34:58.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:58 smithi179 ceph-mon[37907]: osdmap e309: 8 total, 8 up, 8 in 2024-03-24T11:34:58.734 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:58.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:58.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:34:58.865 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:58.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:34:58.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:59.021 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:59.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:59.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:34:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:59.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:59.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:34:59.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:34:59.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:34:59.306 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:59.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:34:59.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:34:59.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[29608]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T11:34:59.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[29608]: pgmap v693: 178 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 500 op/s; 365/279876 objects degraded (0.130%); 20228/279876 objects misplaced (7.227%); 15 MiB/s, 235 objects/s recovering 2024-03-24T11:34:59.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 8304/285612 objects degraded (2.907%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T11:34:59.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[29608]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T11:34:59.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[37907]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T11:34:59.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[37907]: pgmap v693: 178 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 500 op/s; 365/279876 objects degraded (0.130%); 20228/279876 objects misplaced (7.227%); 15 MiB/s, 235 objects/s recovering 2024-03-24T11:34:59.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 8304/285612 objects degraded (2.907%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T11:34:59.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:34:59 smithi179 ceph-mon[37907]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T11:34:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:59.475 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:59.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:34:59.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:34:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:59.637 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:59.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:34:59.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:59 smithi181 ceph-mon[33998]: osdmap e310: 8 total, 8 up, 8 in 2024-03-24T11:34:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:59 smithi181 ceph-mon[33998]: pgmap v693: 178 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 500 op/s; 365/279876 objects degraded (0.130%); 20228/279876 objects misplaced (7.227%); 15 MiB/s, 235 objects/s recovering 2024-03-24T11:34:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:59 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 8304/285612 objects degraded (2.907%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T11:34:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:34:59 smithi181 ceph-mon[33998]: osdmap e311: 8 total, 8 up, 8 in 2024-03-24T11:34:59.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:34:59.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:34:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:59.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:34:59.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:34:59.899 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:34:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:59.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:00.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:00.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:00.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:00.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:00.175 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:00.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:00.262 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:00.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:00.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:00.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:00.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:00.440 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:00.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:00.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:00.587 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:00.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:00.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:00.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:00.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:00.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:00.853 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:00.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:00.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.002 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:01.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:01.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:01.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:01.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:01.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:01.366 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:01.411 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:01.514 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:01.563 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:01.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:01.671 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:35:01.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:01 smithi179 ceph-mon[29608]: pgmap v695: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 809 op/s; 365/275685 objects degraded (0.132%); 19648/275685 objects misplaced (7.127%); 19 MiB/s, 307 objects/s recovering 2024-03-24T11:35:01.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:01 smithi179 ceph-mon[37907]: pgmap v695: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 809 op/s; 365/275685 objects degraded (0.132%); 19648/275685 objects misplaced (7.127%); 19 MiB/s, 307 objects/s recovering 2024-03-24T11:35:01.693 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:01.713 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:35:01.757 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:35:01.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:01.800 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:35:01.839 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:35:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.856 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:01.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:01.876 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:35:01.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:01 smithi181 ceph-mon[33998]: pgmap v695: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 809 op/s; 365/275685 objects degraded (0.132%); 19648/275685 objects misplaced (7.127%); 19 MiB/s, 307 objects/s recovering 2024-03-24T11:35:01.918 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:35:01.957 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:35:01.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:01.969 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:35:01.969 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:35:01.969 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:35:01.969 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:35:01.970 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:35:01.970 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:35:01.970 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:35:01.970 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:35:01.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:01.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:02.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:02.119 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:02.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:02.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:02.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:02.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:02.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:02.379 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:02.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:02.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:02.512 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:02.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:02.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:02.663 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:02.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:35:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:35:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[29608]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T11:35:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3112489516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:35:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:35:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[37907]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T11:35:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3112489516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:02 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:02.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:02.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:02.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:02.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:02.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:02 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:35:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:02 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:35:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:02 smithi181 ceph-mon[33998]: osdmap e312: 8 total, 8 up, 8 in 2024-03-24T11:35:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:02 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3112489516' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:02 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:02.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:02.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:03.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:03.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:03.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:03.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:03.216 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:03.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:03.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:03.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:03.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:03.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:03.451 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:03.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:03.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:03.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:03.621 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:03.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[29608]: pgmap v697: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1007 B/s rd, 0 B/s wr, 651 op/s; 365/275685 objects degraded (0.132%); 19648/275685 objects misplaced (7.127%); 15 MiB/s, 247 objects/s recovering 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[37907]: pgmap v697: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1007 B/s rd, 0 B/s wr, 651 op/s; 365/275685 objects degraded (0.132%); 19648/275685 objects misplaced (7.127%); 15 MiB/s, 247 objects/s recovering 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:03.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:03 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:03.724 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:03.769 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:03.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:03.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:03.878 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:03.899 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:03.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:03 smithi181 ceph-mon[33998]: pgmap v697: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1007 B/s rd, 0 B/s wr, 651 op/s; 365/275685 objects degraded (0.132%); 19648/275685 objects misplaced (7.127%); 15 MiB/s, 247 objects/s recovering 2024-03-24T11:35:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:35:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:03 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:35:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:03 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:04.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:04.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:04.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:04.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:04.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:04.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:04.400 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:04.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:35:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:35:04.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:04.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:04.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:04.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 365/275685 objects degraded (0.132%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:04.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[29608]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T11:35:04.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1962856126' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:35:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 365/275685 objects degraded (0.132%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[37907]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T11:35:04.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1962856126' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:35:04.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:04.817 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:04.859 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:04.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:04.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:04 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 365/275685 objects degraded (0.132%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:04 smithi181 ceph-mon[33998]: osdmap e313: 8 total, 8 up, 8 in 2024-03-24T11:35:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1962856126' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T11:35:04.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:04.992 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:04.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:05.017 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:35:05.094 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:05.138 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:05.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:05.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:05.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:05.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:05.279 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 1 pg_num to 11 2024-03-24T11:35:05.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:05.376 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:35:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:35:05.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:05.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:05.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:05.531 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:05.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:05.633 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:05.680 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:05.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[29608]: pgmap v699: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 912 op/s; 365/263451 objects degraded (0.139%); 17337/263451 objects misplaced (6.581%); 17 MiB/s, 272 objects/s recovering 2024-03-24T11:35:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1962856126' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:35:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[29608]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T11:35:05.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[37907]: pgmap v699: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 912 op/s; 365/263451 objects degraded (0.139%); 17337/263451 objects misplaced (6.581%); 17 MiB/s, 272 objects/s recovering 2024-03-24T11:35:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1962856126' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:35:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:05 smithi179 ceph-mon[37907]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T11:35:05.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:05.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:05.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:05 smithi181 ceph-mon[33998]: pgmap v699: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 912 op/s; 365/263451 objects degraded (0.139%); 17337/263451 objects misplaced (6.581%); 17 MiB/s, 272 objects/s recovering 2024-03-24T11:35:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:05 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1962856126' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T11:35:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:05 smithi181 ceph-mon[33998]: osdmap e314: 8 total, 8 up, 8 in 2024-03-24T11:35:05.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:05.997 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:05.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:06.099 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:06.141 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:06.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:06.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:06.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:06.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:06.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:06.411 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:06.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:06.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:06.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:06.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:06.656 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:06 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:06.656 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:06 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:06.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:06 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:06.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:06 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:06.698 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:06.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:06.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:06.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:06.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:06 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:06 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:06.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:07.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:07.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:07.105 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:07.148 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:07.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:07.211 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:35:07.211 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:35:07.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:29142 8.f e161/150 8:f192749b:::benchmark_data_smithi179_105658_object29141:head v 161'1773, mlcod=161'1770)", 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.766918+0000", 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.44150256399996, 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15131330700000001, 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.212 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766918+0000", 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766918+0000", 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766919+0000", 2024-03-24T11:35:07.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5700000000000002e-07 2024-03-24T11:35:07.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766942+0000", 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3598999999999999e-05 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766953+0000", 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0560000000000001e-05 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.766960+0000", 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3780000000000003e-06 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767036+0000", 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6199999999999995e-05 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.767057+0000", 2024-03-24T11:35:07.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0818e-05 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774194+0000", 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0071372750000000002 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918231+0000", 2024-03-24T11:35:07.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14403682000000001 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:29152 8.0 e161/150 8:05cda426:::benchmark_data_smithi179_105658_object29151:head v 161'1871, mlcod=161'1869)", 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.768057+0000", 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.44036354100001, 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150434658, 2024-03-24T11:35:07.218 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768057+0000", 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768057+0000", 2024-03-24T11:35:07.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768058+0000", 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.71e-07 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768085+0000", 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6781e-05 2024-03-24T11:35:07.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768094+0000", 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3549999999999994e-06 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768100+0000", 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0129999999999999e-06 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768155+0000", 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4916000000000001e-05 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.768173+0000", 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.827e-05 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918454+0000", 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15028112299999999 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918492+0000", 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7329000000000001e-05 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.223 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:29149 8.f e161/150 8:fb6565ee:::benchmark_data_smithi179_105658_object29148:head v 161'1774, mlcod=161'1770)", 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.771191+0000", 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43722918399999, 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14707784800000001, 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771191+0000", 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.224 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771191+0000", 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771192+0000", 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3699999999999999e-07 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771219+0000", 2024-03-24T11:35:07.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.7024000000000001e-05 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771231+0000", 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1865000000000001e-05 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771239+0000", 2024-03-24T11:35:07.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2409999999999995e-06 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771348+0000", 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010856 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771366+0000", 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8156e-05 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918238+0000", 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14687225200000001 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918269+0000", 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0913e-05 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.229 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:29158 8.0 e161/150 8:0bc0922d:::benchmark_data_smithi179_105658_object29157:head v 161'1872, mlcod=161'1869)", 2024-03-24T11:35:07.229 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.771260+0000", 2024-03-24T11:35:07.229 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43716089400004, 2024-03-24T11:35:07.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14730438800000001, 2024-03-24T11:35:07.229 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.229 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.230 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771260+0000", 2024-03-24T11:35:07.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771260+0000", 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771260+0000", 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.22e-07 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771287+0000", 2024-03-24T11:35:07.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.6519000000000001e-05 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771294+0000", 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.8359999999999992e-06 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771300+0000", 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5940000000000001e-06 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771818+0000", 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00051783899999999997 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.771837+0000", 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8904000000000001e-05 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918497+0000", 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14666052299999999 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918564+0000", 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6851000000000001e-05 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:29164 8.0 e161/150 8:0915a6c1:::benchmark_data_smithi179_105658_object29163:head v 161'1873, mlcod=161'1869)", 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.772221+0000", 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43619944500006, 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.146369679, 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772221+0000", 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772221+0000", 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772222+0000", 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3799999999999996e-07 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772243+0000", 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0664000000000001e-05 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772253+0000", 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0314e-05 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772259+0000", 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6420000000000001e-06 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.773257+0000", 2024-03-24T11:35:07.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00099781200000000005 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.773274+0000", 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7127000000000001e-05 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918566+0000", 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.145291789 2024-03-24T11:35:07.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918591+0000", 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4593e-05 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:29163 8.f e161/150 8:fe0437c8:::benchmark_data_smithi179_105658_object29162:head v 161'1775, mlcod=161'1770)", 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.772272+0000", 2024-03-24T11:35:07.240 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43614859800005, 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14602579600000001, 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772272+0000", 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772272+0000", 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772272+0000", 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.0100000000000001e-07 2024-03-24T11:35:07.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772293+0000", 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0262999999999999e-05 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772299+0000", 2024-03-24T11:35:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3430000000000001e-06 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772303+0000", 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4870000000000004e-06 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772849+0000", 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054557499999999997 2024-03-24T11:35:07.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772956+0000", 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107129 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918273+0000", 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14531651100000001 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918298+0000", 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5187000000000001e-05 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29166 8.3 8:ca0fa01c:::benchmark_data_smithi179_105658_object29165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.772858+0000", 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43556241600004, 2024-03-24T11:35:07.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14917075199999999, 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29166 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772858+0000", 2024-03-24T11:35:07.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772858+0000", 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772860+0000", 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3960000000000001e-06 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772889+0000", 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.9328000000000001e-05 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772902+0000", 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2704000000000001e-05 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772909+0000", 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7510000000000008e-06 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.773927+0000", 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0010176969999999999 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774033+0000", 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000106258 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774181+0000", 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000147324 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.921892+0000", 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14771095100000001 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.921903+0000", 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1549000000000001e-05 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.921935+0000", 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1544000000000002e-05 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.921987+0000", 2024-03-24T11:35:07.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2389000000000003e-05 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.922029+0000", 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1860999999999998e-05 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.253 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29167 8.a 8:5a4e4427:::benchmark_data_smithi179_105658_object29166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.772931+0000", 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43548972999997, 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15419401599999999, 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29167 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.254 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772931+0000", 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772931+0000", 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772931+0000", 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4199999999999996e-07 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772952+0000", 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0194e-05 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772959+0000", 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.593e-06 2024-03-24T11:35:07.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.772965+0000", 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.297e-06 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774088+0000", 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00112236 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918849+0000", 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.144760842 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.919481+0000", 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00063230800000000005 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.923793+0000", 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0043116810000000004 2024-03-24T11:35:07.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.923844+0000", 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1199000000000002e-05 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.926975+0000", 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031309110000000001 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.927077+0000", 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000102673 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.927125+0000", 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7416000000000001e-05 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:29169 8.3 8:c6a598cc:::benchmark_data_smithi179_105658_object29168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.774109+0000", 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43431135499998, 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15326916800000001, 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 29169 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774109+0000", 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774109+0000", 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774110+0000", 2024-03-24T11:35:07.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.218e-06 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.774200+0000", 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9511000000000004e-05 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918110+0000", 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.143910027 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918142+0000", 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1829000000000003e-05 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920165+0000", 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020231419999999999 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920275+0000", 2024-03-24T11:35:07.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010989000000000001 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 5,6", 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920746+0000", 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00047066799999999999 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.926147+0000", 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054015390000000003 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.926159+0000", 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1462e-05 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.927110+0000", 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00095176500000000003 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.927306+0000", 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00019609099999999999 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.927378+0000", 2024-03-24T11:35:07.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.2026000000000005e-05 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:29174 8.b e161/150 8:d5efc351:::benchmark_data_smithi179_105658_object29173:head v 161'1835, mlcod=161'1833)", 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:25:25.777122+0000", 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 581.43129905000001, 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14907120500000001, 2024-03-24T11:35:07.268 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.777122+0000", 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.777122+0000", 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.777123+0000", 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2729999999999999e-06 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.777149+0000", 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.5880999999999998e-05 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918675+0000", 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.141526178 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.918681+0000", 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5130000000000002e-06 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920104+0000", 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001423108 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.920124+0000", 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9072999999999998e-05 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.926160+0000", 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060368610000000001 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:25:25.926193+0000", 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.2317999999999998e-05 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.273 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127643 8.7 8:e7090b63:::benchmark_data_smithi179_105658_object127642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.433572+0000", 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.774848185, 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16173941999999999, 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127643 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.433572+0000", 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.433572+0000", 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.433574+0000", 2024-03-24T11:35:07.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.494e-06 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.433599+0000", 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4688000000000001e-05 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.433613+0000", 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4419e-05 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.433623+0000", 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8079999999999999e-06 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.440679+0000", 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0070560880000000003 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.440805+0000", 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125757 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.440960+0000", 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00015561700000000001 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.447097+0000", 2024-03-24T11:35:07.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006136351 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.447138+0000", 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0951999999999997e-05 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.595065+0000", 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14792719600000001 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:07.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.595251+0000", 2024-03-24T11:35:07.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018651200000000001 2024-03-24T11:35:07.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.281 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.281 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.595312+0000", 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0538000000000002e-05 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop_reply(client.24959.0:127643 8.7 e176/150 ondisk, result = 0)", 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.451597+0000", 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.75682349800002, 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14372143300000001, 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.282 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "started", 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.451597+0000", 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.451597+0000", 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.451598+0000", 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5499999999999998e-07 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.451602+0000", 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7690000000000004e-06 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.451611+0000", 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7399999999999993e-06 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.451618+0000", 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3560000000000004e-06 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.595052+0000", 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14343304200000001 2024-03-24T11:35:07.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.595059+0000", 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5179999999999998e-06 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.595319+0000", 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025945299999999998 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.286 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127667 8.7 8:e566e9c6:::benchmark_data_smithi179_105658_object127666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.453370+0000", 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.75505076399998, 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.165078066, 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127667 2024-03-24T11:35:07.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.453370+0000", 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.453370+0000", 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.453371+0000", 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.175e-06 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.453419+0000", 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8223999999999998e-05 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.453431+0000", 2024-03-24T11:35:07.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1552999999999999e-05 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.453438+0000", 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5379999999999997e-06 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.601473+0000", 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14803486799999999 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.601627+0000", 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000153824 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.601755+0000", 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000127818 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.610577+0000", 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0088218150000000002 2024-03-24T11:35:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.610665+0000", 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8177999999999995e-05 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618313+0000", 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0076478750000000002 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618370+0000", 2024-03-24T11:35:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7821000000000001e-05 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618448+0000", 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7377000000000001e-05 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127669 8.7 8:e7fc8cd2:::benchmark_data_smithi179_105658_object127668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:35:07.293 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.454241+0000", 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.754179507, 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16448543800000001, 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127669 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454241+0000", 2024-03-24T11:35:07.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.318 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454241+0000", 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454242+0000", 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6899999999999996e-07 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454291+0000", 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8677999999999997e-05 2024-03-24T11:35:07.319 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454303+0000", 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1793e-05 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454311+0000", 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0870000000000007e-06 2024-03-24T11:35:07.320 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604288+0000", 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14997727599999999 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604430+0000", 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014252500000000001 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:35:07.321 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604613+0000", 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018291299999999999 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.610624+0000", 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060103600000000002 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.322 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.617537+0000", 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0069134740000000002 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618500+0000", 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00096323100000000005 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.323 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618601+0000", 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000100735 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618727+0000", 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012539699999999999 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.324 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:127665 8.f e176/150 8:ff943aea:::benchmark_data_smithi179_105658_object127664:head v 176'8032, mlcod=176'8030)", 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.454354+0000", 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.75406620899997, 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14989934299999999, 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.325 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454354+0000", 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454354+0000", 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.326 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454355+0000", 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9800000000000003e-07 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454379+0000", 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4026999999999999e-05 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454389+0000", 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.8239999999999995e-06 2024-03-24T11:35:07.327 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.454395+0000", 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.579e-06 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.594904+0000", 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14050842399999999 2024-03-24T11:35:07.328 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.594922+0000", 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8360999999999999e-05 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604208+0000", 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0092860700000000004 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.604254+0000", 2024-03-24T11:35:07.329 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5359999999999999e-05 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:127666 8.f e176/150 8:f55f5ea7:::benchmark_data_smithi179_105658_object127665:head v 176'8033, mlcod=176'8030)", 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.458221+0000", 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.75020003200001, 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14934468400000001, 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.330 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.458221+0000", 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.458221+0000", 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.458221+0000", 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0100000000000004e-07 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.458246+0000", 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4538000000000001e-05 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.332 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.458255+0000", 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.5889999999999992e-06 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.458263+0000", 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3089999999999998e-06 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.333 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.599267+0000", 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14100445 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.599292+0000", 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4723e-05 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.607153+0000", 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0078606059999999991 2024-03-24T11:35:07.334 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.607565+0000", 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00041266799999999999 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127678 8.7 8:e7b5aad8:::benchmark_data_smithi179_105658_object127677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.463180+0000", 2024-03-24T11:35:07.335 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.745240846, 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15856587999999999, 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127678 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.336 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463180+0000", 2024-03-24T11:35:07.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.337 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.337 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.337 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463180+0000", 2024-03-24T11:35:07.337 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463181+0000", 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5939999999999999e-06 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463204+0000", 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.2997e-05 2024-03-24T11:35:07.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463217+0000", 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2259000000000001e-05 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463225+0000", 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4789999999999996e-06 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.339 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.607227+0000", 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14400195599999999 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.607356+0000", 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000128658 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.340 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.607529+0000", 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000173308 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.610709+0000", 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031802129999999999 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.341 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.617575+0000", 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006866189 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.621647+0000", 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040713709999999998 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.621694+0000", 2024-03-24T11:35:07.342 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6842999999999999e-05 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.621746+0000", 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2012999999999998e-05 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:127679 8.7 8:e299e6c8:::benchmark_data_smithi179_105658_object127678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.463239+0000", 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.74518126700002, 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.160262973, 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 127679 2024-03-24T11:35:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463239+0000", 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463239+0000", 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463240+0000", 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1900000000000003e-07 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463260+0000", 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9922e-05 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463269+0000", 2024-03-24T11:35:07.346 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0480000000000004e-06 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.463275+0000", 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0249999999999999e-06 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.610743+0000", 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14746864200000001 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.347 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.610875+0000", 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013195000000000001 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,6", 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.611133+0000", 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025782499999999999 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.348 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.617620+0000", 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0064864429999999997 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.621753+0000", 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0041331680000000004 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.623404+0000", 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016510959999999999 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.349 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.623462+0000", 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8227999999999999e-05 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.623502+0000", 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0107000000000001e-05 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.24959.0:127676 8.f e176/150 8:fcc02684:::benchmark_data_smithi179_105658_object127675:head v 176'8034, mlcod=176'8030)", 2024-03-24T11:35:07.350 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:26:43.465028+0000", 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 503.74339233299997, 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15346336199999999, 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.465028+0000", 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.351 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.465028+0000", 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.465029+0000", 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2500000000000004e-07 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.352 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.465054+0000", 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.4851999999999999e-05 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.593566+0000", 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12851231799999999 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.353 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.593579+0000", 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2388999999999999e-05 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.601548+0000", 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0079689059999999996 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.354 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.601607+0000", 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9487999999999997e-05 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618452+0000", 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.016844929000000002 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:26:43.618492+0000", 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9654999999999999e-05 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.355 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.24959.0:248270 8.a 8:5a620225:::benchmark_data_smithi179_105658_object120228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:28:28.507255+0000", 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 398.70116557099999, 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15119297100000001, 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:35:07.356 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.24959", 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/768340263", 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 248270 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:35:07.357 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.507255+0000", 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.507255+0000", 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.507256+0000", 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3149999999999999e-06 2024-03-24T11:35:07.358 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.507261+0000", 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4669999999999998e-06 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.507272+0000", 2024-03-24T11:35:07.359 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1619e-05 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.507280+0000", 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4630000000000004e-06 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.509130+0000", 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001849774 2024-03-24T11:35:07.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.509255+0000", 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012551399999999999 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.509379+0000", 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123378 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.512554+0000", 2024-03-24T11:35:07.361 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031752289999999999 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.542751+0000", 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030197491999999999 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.658276+0000", 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.115524792 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.362 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.658377+0000", 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000101241 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:28:28.658448+0000", 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0686999999999996e-05 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:35:07.363 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:35:07.407 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:07.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:07.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:07.465 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:35:07.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:07.539 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:07.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:07.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: pgmap v701: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 700 op/s; 365/263076 objects degraded (0.139%); 17337/263076 objects misplaced (6.590%); 14 MiB/s, 224 objects/s recovering 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4198221490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:35:07.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[29608]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: pgmap v701: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 700 op/s; 365/263076 objects degraded (0.139%); 17337/263076 objects misplaced (6.590%); 14 MiB/s, 224 objects/s recovering 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4198221490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:35:07.681 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:07 smithi179 ceph-mon[37907]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T11:35:07.687 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:07.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:07.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:07.825 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:07.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: pgmap v701: 178 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 700 op/s; 365/263076 objects degraded (0.139%); 17337/263076 objects misplaced (6.590%); 14 MiB/s, 224 objects/s recovering 2024-03-24T11:35:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T11:35:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: osdmap e315: 8 total, 8 up, 8 in 2024-03-24T11:35:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4198221490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:35:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:35:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:35:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:35:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:07 smithi181 ceph-mon[33998]: osdmap e316: 8 total, 8 up, 8 in 2024-03-24T11:35:07.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:07.971 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:07.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:08.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:08.117 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:08.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:08.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:08.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:08.273 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:08.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:08.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:08.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:08.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:08.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:08.574 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:08.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:08.624 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:08.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:08.736 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:08.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:08.837 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:08.884 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:08.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:08.986 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:09.031 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:09.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:09.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:35:09.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:09.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:09.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:09.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:09.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:09.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:09.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:09.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:09.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[29608]: pgmap v704: 178 pgs: 1 active+remapped+backfilling, 5 remapped+peering, 3 active+recovering+undersized+remapped, 167 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 4.6 KiB/s rd, 27 KiB/s wr, 615 op/s; 365/255930 objects degraded (0.143%); 15102/255930 objects misplaced (5.901%); 20 MiB/s, 324 objects/s recovering 2024-03-24T11:35:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[29608]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T11:35:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[29608]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:35:09.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 365/255930 objects degraded (0.143%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:09.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[29608]: osdmap e318: 8 total, 8 up, 8 in 2024-03-24T11:35:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[37907]: pgmap v704: 178 pgs: 1 active+remapped+backfilling, 5 remapped+peering, 3 active+recovering+undersized+remapped, 167 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 4.6 KiB/s rd, 27 KiB/s wr, 615 op/s; 365/255930 objects degraded (0.143%); 15102/255930 objects misplaced (5.901%); 20 MiB/s, 324 objects/s recovering 2024-03-24T11:35:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[37907]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T11:35:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[37907]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:35:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 365/255930 objects degraded (0.143%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:09.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:09 smithi179 ceph-mon[37907]: osdmap e318: 8 total, 8 up, 8 in 2024-03-24T11:35:09.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:09.836 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:09 smithi181 ceph-mon[33998]: pgmap v704: 178 pgs: 1 active+remapped+backfilling, 5 remapped+peering, 3 active+recovering+undersized+remapped, 167 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 4.6 KiB/s rd, 27 KiB/s wr, 615 op/s; 365/255930 objects degraded (0.143%); 15102/255930 objects misplaced (5.901%); 20 MiB/s, 324 objects/s recovering 2024-03-24T11:35:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:09 smithi181 ceph-mon[33998]: osdmap e317: 8 total, 8 up, 8 in 2024-03-24T11:35:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:09 smithi181 ceph-mon[33998]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-24T11:35:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:09 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 365/255930 objects degraded (0.143%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:09 smithi181 ceph-mon[33998]: osdmap e318: 8 total, 8 up, 8 in 2024-03-24T11:35:09.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:09.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:09.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:10.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:10.099 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:10.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:10.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:10.235 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:10.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:10.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:10.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:10.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:10.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:10.525 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:10.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:10.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:10.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:35:10.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:10.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:10.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:10.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:10.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:10.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:10.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:10.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:11.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:11.083 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:11.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:11.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:11.217 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:11.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:11.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:11.366 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:11.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:11.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:11.492 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:11.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:11.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:11.642 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:11.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:11 smithi181 ceph-mon[33998]: osdmap e319: 8 total, 8 up, 8 in 2024-03-24T11:35:11.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:11 smithi181 ceph-mon[33998]: pgmap v708: 178 pgs: 2 peering, 1 active+remapped+backfilling, 4 remapped+peering, 3 active+recovering+undersized+remapped, 1 active+remapped, 163 active+clean, 2 active+undersized+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 4.8 KiB/s rd, 28 KiB/s wr, 1.10k op/s; 365/250206 objects degraded (0.146%); 15102/250206 objects misplaced (6.036%); 21 MiB/s, 340 objects/s recovering 2024-03-24T11:35:11.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:11 smithi179 ceph-mon[29608]: osdmap e319: 8 total, 8 up, 8 in 2024-03-24T11:35:11.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:11 smithi179 ceph-mon[29608]: pgmap v708: 178 pgs: 2 peering, 1 active+remapped+backfilling, 4 remapped+peering, 3 active+recovering+undersized+remapped, 1 active+remapped, 163 active+clean, 2 active+undersized+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 4.8 KiB/s rd, 28 KiB/s wr, 1.10k op/s; 365/250206 objects degraded (0.146%); 15102/250206 objects misplaced (6.036%); 21 MiB/s, 340 objects/s recovering 2024-03-24T11:35:11.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:11 smithi179 ceph-mon[37907]: osdmap e319: 8 total, 8 up, 8 in 2024-03-24T11:35:11.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:11 smithi179 ceph-mon[37907]: pgmap v708: 178 pgs: 2 peering, 1 active+remapped+backfilling, 4 remapped+peering, 3 active+recovering+undersized+remapped, 1 active+remapped, 163 active+clean, 2 active+undersized+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 4.8 KiB/s rd, 28 KiB/s wr, 1.10k op/s; 365/250206 objects degraded (0.146%); 15102/250206 objects misplaced (6.036%); 21 MiB/s, 340 objects/s recovering 2024-03-24T11:35:11.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:11.820 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:11.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:11.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:11.954 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:11.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:12.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:12.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:12.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:12.192 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:12.216 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:12.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:12.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:12.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:12.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:12.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:12.482 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:12.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:12.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:12.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:12.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:12.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:12.736 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:35:12.736 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:35:12.736 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-24T11:35:12.736 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T11:35:12.736 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T11:35:12.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-24T11:35:12.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:12.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:12.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:12.933 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:12.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:13.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:13.099 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:13.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:13.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:13.240 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:13.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:13.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:13.392 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:13.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:13.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:13.534 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:13.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:13.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:13.679 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:13.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:13 smithi179 ceph-mon[29608]: pgmap v709: 178 pgs: 2 peering, 1 active+remapped+backfilling, 4 remapped+peering, 3 active+recovering+undersized+remapped, 1 active+remapped, 163 active+clean, 2 active+undersized+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 3.9 KiB/s rd, 22 KiB/s wr, 877 op/s; 365/250206 objects degraded (0.146%); 15102/250206 objects misplaced (6.036%); 17 MiB/s, 271 objects/s recovering 2024-03-24T11:35:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:13 smithi179 ceph-mon[37907]: pgmap v709: 178 pgs: 2 peering, 1 active+remapped+backfilling, 4 remapped+peering, 3 active+recovering+undersized+remapped, 1 active+remapped, 163 active+clean, 2 active+undersized+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 3.9 KiB/s rd, 22 KiB/s wr, 877 op/s; 365/250206 objects degraded (0.146%); 15102/250206 objects misplaced (6.036%); 17 MiB/s, 271 objects/s recovering 2024-03-24T11:35:13.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:13.830 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:13.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:13 smithi181 ceph-mon[33998]: pgmap v709: 178 pgs: 2 peering, 1 active+remapped+backfilling, 4 remapped+peering, 3 active+recovering+undersized+remapped, 1 active+remapped, 163 active+clean, 2 active+undersized+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 3.9 KiB/s rd, 22 KiB/s wr, 877 op/s; 365/250206 objects degraded (0.146%); 15102/250206 objects misplaced (6.036%); 17 MiB/s, 271 objects/s recovering 2024-03-24T11:35:13.931 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:14.017 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:14.020 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:14.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:14.046 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:14.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:14.187 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:14.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:14.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:14.320 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:14.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:14.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:35:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:35:14.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:14.461 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:14.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:14.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:14.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:14.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:14.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:14.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:14 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 365/250206 objects degraded (0.146%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:14.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:14 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 365/250206 objects degraded (0.146%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:14.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:14.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:14.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:14.864 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:14.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:14.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:14 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 365/250206 objects degraded (0.146%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:14.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.013 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:15.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:15.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:15.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:15.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:15.291 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:15.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:15.426 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:35:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:35:15.427 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:15.529 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:15.577 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:15.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[37907]: pgmap v710: 178 pgs: 1 active+remapped+backfill_toofull, 2 peering, 3 active+recovering+undersized+remapped, 170 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 KiB/s wr, 1.01k op/s; 365/237861 objects degraded (0.153%); 13257/237861 objects misplaced (5.573%); 9.8 MiB/s, 155 objects/s recovering 2024-03-24T11:35:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[37907]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:35:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[37907]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:35:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[29608]: pgmap v710: 178 pgs: 1 active+remapped+backfill_toofull, 2 peering, 3 active+recovering+undersized+remapped, 170 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 KiB/s wr, 1.01k op/s; 365/237861 objects degraded (0.153%); 13257/237861 objects misplaced (5.573%); 9.8 MiB/s, 155 objects/s recovering 2024-03-24T11:35:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[29608]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:35:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:15 smithi179 ceph-mon[29608]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:35:15.724 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:15.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:15.849 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:15 smithi181 ceph-mon[33998]: pgmap v710: 178 pgs: 1 active+remapped+backfill_toofull, 2 peering, 3 active+recovering+undersized+remapped, 170 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 KiB/s wr, 1.01k op/s; 365/237861 objects degraded (0.153%); 13257/237861 objects misplaced (5.573%); 9.8 MiB/s, 155 objects/s recovering 2024-03-24T11:35:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:15 smithi181 ceph-mon[33998]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:35:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:15 smithi181 ceph-mon[33998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-24T11:35:15.951 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:15.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:15.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:16.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:16.127 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:16.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:16.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:16.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:16.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:16.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:16.422 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:16.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:16.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:16.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:16.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:16.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:16.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:16.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:16.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:16 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T11:35:16.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:16 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T11:35:16.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:16 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T11:35:16.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:16 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T11:35:16.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:16.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:16.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:16 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T11:35:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:16 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-24T11:35:16.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:16.949 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:16.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:17.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:17.082 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:17.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:17.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:17.217 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:17.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:17.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:17.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:17.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:17.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:17.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:17.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:17.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:17.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:17.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:17.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[37907]: pgmap v711: 178 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 2.4 KiB/s rd, 14 KiB/s wr, 780 op/s; 365/237459 objects degraded (0.154%); 13257/237459 objects misplaced (5.583%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T11:35:17.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-24T11:35:17.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[37907]: osdmap e320: 8 total, 8 up, 8 in 2024-03-24T11:35:17.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[37907]: osdmap e321: 8 total, 8 up, 8 in 2024-03-24T11:35:17.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[29608]: pgmap v711: 178 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 2.4 KiB/s rd, 14 KiB/s wr, 780 op/s; 365/237459 objects degraded (0.154%); 13257/237459 objects misplaced (5.583%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T11:35:17.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-24T11:35:17.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[29608]: osdmap e320: 8 total, 8 up, 8 in 2024-03-24T11:35:17.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:17 smithi179 ceph-mon[29608]: osdmap e321: 8 total, 8 up, 8 in 2024-03-24T11:35:17.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:17.755 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:17.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:17.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:17.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:17.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:17 smithi181 ceph-mon[33998]: pgmap v711: 178 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 2.4 KiB/s rd, 14 KiB/s wr, 780 op/s; 365/237459 objects degraded (0.154%); 13257/237459 objects misplaced (5.583%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T11:35:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:17 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-24T11:35:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:17 smithi181 ceph-mon[33998]: osdmap e320: 8 total, 8 up, 8 in 2024-03-24T11:35:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:17 smithi181 ceph-mon[33998]: osdmap e321: 8 total, 8 up, 8 in 2024-03-24T11:35:17.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:18.031 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:18.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:18.133 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:18.172 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:18.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:18.274 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:18.323 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:18.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:18.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:18.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:18.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:18.552 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:18.604 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:18.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:18.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:18.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:18.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:18.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:18.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:18.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:18.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:19.012 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:19.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:19.156 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:19.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:19.288 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:19.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:19.413 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:19.514 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:19.535 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[29608]: pgmap v714: 178 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 850 op/s; 372/229869 objects degraded (0.162%); 11763/229869 objects misplaced (5.117%); 12 MiB/s, 189 objects/s recovering 2024-03-24T11:35:19.536 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[37907]: pgmap v714: 178 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 850 op/s; 372/229869 objects degraded (0.162%); 11763/229869 objects misplaced (5.117%); 12 MiB/s, 189 objects/s recovering 2024-03-24T11:35:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.546 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:19.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:19.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.687 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:19.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:19.788 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.827 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:19.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:19 smithi181 ceph-mon[33998]: pgmap v714: 178 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 172 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 3.0 KiB/s rd, 14 KiB/s wr, 850 op/s; 372/229869 objects degraded (0.162%); 11763/229869 objects misplaced (5.117%); 12 MiB/s, 189 objects/s recovering 2024-03-24T11:35:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:19 smithi181 ceph-mon[33998]: osdmap e322: 8 total, 8 up, 8 in 2024-03-24T11:35:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/711370276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:19 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/229869 objects degraded (0.162%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:19.929 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[29608]: osdmap e322: 8 total, 8 up, 8 in 2024-03-24T11:35:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/711370276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/229869 objects degraded (0.162%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[37907]: osdmap e322: 8 total, 8 up, 8 in 2024-03-24T11:35:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/711370276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:19 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/229869 objects degraded (0.162%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:19.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:19.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:20.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:20.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:20.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:20.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:20.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:20.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:20.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:20.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:20.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:20.476 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:20.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:20.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:20.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:20.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:20.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:20.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:20 smithi179 ceph-mon[29608]: osdmap e323: 8 total, 8 up, 8 in 2024-03-24T11:35:20.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:20 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T11:35:20.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:20 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T11:35:20.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:20 smithi179 ceph-mon[37907]: osdmap e323: 8 total, 8 up, 8 in 2024-03-24T11:35:20.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:20 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T11:35:20.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:20 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T11:35:20.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:20.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:20.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:20.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:20.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:20.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:20 smithi181 ceph-mon[33998]: osdmap e323: 8 total, 8 up, 8 in 2024-03-24T11:35:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:20 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T11:35:20.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:20 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-24T11:35:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:20.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:20.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:21.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:21.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:21.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:21.146 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:35:21.147 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:35:21.148 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:35:21.148 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:35:21.148 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:35:21.148 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:35:21.148 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:35:21.162 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:21.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:21.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:21.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:21.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:21.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:21.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:21.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:21.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:21.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:21.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:21.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:21.707 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:21.730 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:21.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:21.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:21.876 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:21.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:21 smithi181 ceph-mon[33998]: pgmap v717: 177 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 372/224232 objects degraded (0.166%); 11763/224232 objects misplaced (5.246%); 9.1 MiB/s, 145 objects/s recovering 2024-03-24T11:35:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:21 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-24T11:35:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:21 smithi181 ceph-mon[33998]: osdmap e324: 8 total, 8 up, 8 in 2024-03-24T11:35:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:21 smithi179 ceph-mon[29608]: pgmap v717: 177 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 372/224232 objects degraded (0.166%); 11763/224232 objects misplaced (5.246%); 9.1 MiB/s, 145 objects/s recovering 2024-03-24T11:35:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:21 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-24T11:35:21.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:21 smithi179 ceph-mon[29608]: osdmap e324: 8 total, 8 up, 8 in 2024-03-24T11:35:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:21 smithi179 ceph-mon[37907]: pgmap v717: 177 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 372/224232 objects degraded (0.166%); 11763/224232 objects misplaced (5.246%); 9.1 MiB/s, 145 objects/s recovering 2024-03-24T11:35:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:21 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-24T11:35:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:21 smithi179 ceph-mon[37907]: osdmap e324: 8 total, 8 up, 8 in 2024-03-24T11:35:21.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:22.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:22.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:22.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:22.166 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:22.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:22.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:22.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:22.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:22.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:22.430 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:22.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:22.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:22.561 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:22.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:22.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:22.706 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:22.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:22.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:22.809 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:22 smithi179 ceph-mon[29608]: osdmap e325: 8 total, 8 up, 8 in 2024-03-24T11:35:22.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:22 smithi179 ceph-mon[37907]: osdmap e325: 8 total, 8 up, 8 in 2024-03-24T11:35:22.853 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:22.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:22 smithi181 ceph-mon[33998]: osdmap e325: 8 total, 8 up, 8 in 2024-03-24T11:35:22.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:23.000 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:23.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:23.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:23.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:23.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:23.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.416 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:23.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:23.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:23.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:23.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.687 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:23.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:23 smithi179 ceph-mon[29608]: pgmap v720: 177 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 469 op/s; 372/224232 objects degraded (0.166%); 11763/224232 objects misplaced (5.246%) 2024-03-24T11:35:23.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:23 smithi179 ceph-mon[29608]: osdmap e326: 8 total, 8 up, 8 in 2024-03-24T11:35:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:23 smithi179 ceph-mon[37907]: pgmap v720: 177 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 469 op/s; 372/224232 objects degraded (0.166%); 11763/224232 objects misplaced (5.246%) 2024-03-24T11:35:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:23 smithi179 ceph-mon[37907]: osdmap e326: 8 total, 8 up, 8 in 2024-03-24T11:35:23.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:23.811 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:23 smithi181 ceph-mon[33998]: pgmap v720: 177 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 171 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 469 op/s; 372/224232 objects degraded (0.166%); 11763/224232 objects misplaced (5.246%) 2024-03-24T11:35:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:23 smithi181 ceph-mon[33998]: osdmap e326: 8 total, 8 up, 8 in 2024-03-24T11:35:23.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:23.955 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:23.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:24.057 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:24.099 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:24.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:24.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:24.225 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:24.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:24.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:24.358 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:24.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:24.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:35:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:35:24.460 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:24.500 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:24.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:24.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:24.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:24.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:24.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:24.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:24.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:24.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:24 smithi181 ceph-mon[33998]: osdmap e327: 8 total, 8 up, 8 in 2024-03-24T11:35:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:24 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/224232 objects degraded (0.166%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:24 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T11:35:24.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:24 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T11:35:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:24.910 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:24.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[29608]: osdmap e327: 8 total, 8 up, 8 in 2024-03-24T11:35:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/224232 objects degraded (0.166%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T11:35:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T11:35:24.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[37907]: osdmap e327: 8 total, 8 up, 8 in 2024-03-24T11:35:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/224232 objects degraded (0.166%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T11:35:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:24 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-24T11:35:25.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:25.052 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:25.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:25.155 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:25.193 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:25.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:25.295 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:25.338 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:25.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:25.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:35:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:35:25.479 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:25.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:25.581 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:25.627 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:25.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:25.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:25.762 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:25.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:25.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:25.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:25.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:25 smithi181 ceph-mon[33998]: pgmap v723: 176 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 372/211674 objects degraded (0.176%); 10060/211674 objects misplaced (4.753%); 10 MiB/s, 167 objects/s recovering 2024-03-24T11:35:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:25 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-24T11:35:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:25 smithi181 ceph-mon[33998]: osdmap e328: 8 total, 8 up, 8 in 2024-03-24T11:35:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[29608]: pgmap v723: 176 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 372/211674 objects degraded (0.176%); 10060/211674 objects misplaced (4.753%); 10 MiB/s, 167 objects/s recovering 2024-03-24T11:35:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-24T11:35:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[29608]: osdmap e328: 8 total, 8 up, 8 in 2024-03-24T11:35:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[37907]: pgmap v723: 176 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 372/211674 objects degraded (0.176%); 10060/211674 objects misplaced (4.753%); 10 MiB/s, 167 objects/s recovering 2024-03-24T11:35:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-24T11:35:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:25 smithi179 ceph-mon[37907]: osdmap e328: 8 total, 8 up, 8 in 2024-03-24T11:35:26.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:26.035 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:26.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:26.136 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:26.174 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:26.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:26.276 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:26.333 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:26.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:26.434 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:26.475 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:26.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:26.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:26.616 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:26.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:26.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:26.742 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:26.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:26.844 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:26.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:26.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:26 smithi181 ceph-mon[33998]: osdmap e329: 8 total, 8 up, 8 in 2024-03-24T11:35:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:26 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:35:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:26 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:35:26.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:26 smithi179 ceph-mon[29608]: osdmap e329: 8 total, 8 up, 8 in 2024-03-24T11:35:26.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:26 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:35:26.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:26 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:35:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:26 smithi179 ceph-mon[37907]: osdmap e329: 8 total, 8 up, 8 in 2024-03-24T11:35:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:26 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:35:26.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:26 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:35:27.002 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:27.033 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:27.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:27.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:27.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:27.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:27.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:27.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:27.327 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:27.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:27.344 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:27.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:27.481 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:27.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:27.583 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:27.631 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:27.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:27.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:27.774 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:27.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:27.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:27 smithi181 ceph-mon[33998]: pgmap v726: 176 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 372/211251 objects degraded (0.176%); 10060/211251 objects misplaced (4.762%); 10 MiB/s, 167 objects/s recovering 2024-03-24T11:35:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:27 smithi181 ceph-mon[33998]: osdmap e330: 8 total, 8 up, 8 in 2024-03-24T11:35:27.921 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:27.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:27 smithi179 ceph-mon[29608]: pgmap v726: 176 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 372/211251 objects degraded (0.176%); 10060/211251 objects misplaced (4.762%); 10 MiB/s, 167 objects/s recovering 2024-03-24T11:35:27.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:27 smithi179 ceph-mon[29608]: osdmap e330: 8 total, 8 up, 8 in 2024-03-24T11:35:27.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:27 smithi179 ceph-mon[37907]: pgmap v726: 176 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 372/211251 objects degraded (0.176%); 10060/211251 objects misplaced (4.762%); 10 MiB/s, 167 objects/s recovering 2024-03-24T11:35:27.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:27 smithi179 ceph-mon[37907]: osdmap e330: 8 total, 8 up, 8 in 2024-03-24T11:35:28.023 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:28.061 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:28.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:28.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:28.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:28.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:28.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:28.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:28.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:28.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:28.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:28.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:28.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:28.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:28.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:28.601 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:28.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:28.702 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:28.743 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:28.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:28.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:28.890 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:28.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:28.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:28 smithi181 ceph-mon[33998]: osdmap e331: 8 total, 8 up, 8 in 2024-03-24T11:35:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:28 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T11:35:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:28 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T11:35:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:28 smithi179 ceph-mon[29608]: osdmap e331: 8 total, 8 up, 8 in 2024-03-24T11:35:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:28 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T11:35:28.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:28 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T11:35:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:28 smithi179 ceph-mon[37907]: osdmap e331: 8 total, 8 up, 8 in 2024-03-24T11:35:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:28 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T11:35:28.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:28 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-24T11:35:28.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:29.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:29.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:29.335 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:29.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:35:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:29.382 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:29.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:29.483 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:29.510 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:29.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:29.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:29.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:29.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[29608]: pgmap v729: 175 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 536 op/s; 372/205203 objects degraded (0.181%); 9095/205203 objects misplaced (4.432%); 6.1 MiB/s, 97 objects/s recovering 2024-03-24T11:35:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-24T11:35:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[29608]: osdmap e332: 8 total, 8 up, 8 in 2024-03-24T11:35:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/205203 objects degraded (0.181%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[37907]: pgmap v729: 175 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 536 op/s; 372/205203 objects degraded (0.181%); 9095/205203 objects misplaced (4.432%); 6.1 MiB/s, 97 objects/s recovering 2024-03-24T11:35:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-24T11:35:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[37907]: osdmap e332: 8 total, 8 up, 8 in 2024-03-24T11:35:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:29 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/205203 objects degraded (0.181%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:29.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:29.772 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:29.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:29.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:29.902 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:35:29 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=sqlstore.transactions t=2024-03-24T11:35:29.823416574Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:35:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:29 smithi181 ceph-mon[33998]: pgmap v729: 175 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 536 op/s; 372/205203 objects degraded (0.181%); 9095/205203 objects misplaced (4.432%); 6.1 MiB/s, 97 objects/s recovering 2024-03-24T11:35:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:29 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-24T11:35:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:29 smithi181 ceph-mon[33998]: osdmap e332: 8 total, 8 up, 8 in 2024-03-24T11:35:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:29 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/205203 objects degraded (0.181%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:29.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:29.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:30.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:30.046 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:30.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.178 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:30.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:30.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:30.329 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:30.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:30.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:30.568 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:30.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:30.707 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:30.732 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:30.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:30.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:30 smithi181 ceph-mon[33998]: osdmap e333: 8 total, 8 up, 8 in 2024-03-24T11:35:30.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:30 smithi179 ceph-mon[29608]: osdmap e333: 8 total, 8 up, 8 in 2024-03-24T11:35:30.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:30 smithi179 ceph-mon[37907]: osdmap e333: 8 total, 8 up, 8 in 2024-03-24T11:35:30.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:30.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:30.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:31.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:31.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:31.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:31.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:31.264 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:31.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:31.365 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:31.410 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:31.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:31.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:31.567 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:31.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:31.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:31.704 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:31.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:31.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:31.834 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:31.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:31.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:31 smithi179 ceph-mon[29608]: pgmap v732: 175 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 372/196665 objects degraded (0.189%); 8205/196665 objects misplaced (4.172%); 11 MiB/s, 181 objects/s recovering 2024-03-24T11:35:31.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:31 smithi179 ceph-mon[29608]: osdmap e334: 8 total, 8 up, 8 in 2024-03-24T11:35:31.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:31 smithi179 ceph-mon[37907]: pgmap v732: 175 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 372/196665 objects degraded (0.189%); 8205/196665 objects misplaced (4.172%); 11 MiB/s, 181 objects/s recovering 2024-03-24T11:35:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:31 smithi179 ceph-mon[37907]: osdmap e334: 8 total, 8 up, 8 in 2024-03-24T11:35:31.975 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:31.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:32.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:32.109 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:32.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:31 smithi181 ceph-mon[33998]: pgmap v732: 175 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 169 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 372/196665 objects degraded (0.189%); 8205/196665 objects misplaced (4.172%); 11 MiB/s, 181 objects/s recovering 2024-03-24T11:35:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:31 smithi181 ceph-mon[33998]: osdmap e334: 8 total, 8 up, 8 in 2024-03-24T11:35:32.211 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:32.254 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:32.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:32.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:32.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:32.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:32.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:32.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:32.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:32.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:32.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:32.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:32.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:32.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:32.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:32.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:32.921 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:32.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:32 smithi179 ceph-mon[29608]: osdmap e335: 8 total, 8 up, 8 in 2024-03-24T11:35:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T11:35:32.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T11:35:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:32 smithi179 ceph-mon[37907]: osdmap e335: 8 total, 8 up, 8 in 2024-03-24T11:35:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T11:35:32.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T11:35:33.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:33.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:33.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:33.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:32 smithi181 ceph-mon[33998]: osdmap e335: 8 total, 8 up, 8 in 2024-03-24T11:35:33.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T11:35:33.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-24T11:35:33.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:33.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:33.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:33.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:33.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:33.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:33.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:33.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:33.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:33.601 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:33.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:33.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:33.737 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:33.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:33.838 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:33.872 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:33.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[29608]: pgmap v735: 174 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 168 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 715 op/s; 372/196665 objects degraded (0.189%); 8205/196665 objects misplaced (4.172%); 5.3 MiB/s, 84 objects/s recovering 2024-03-24T11:35:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-24T11:35:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[29608]: osdmap e336: 8 total, 8 up, 8 in 2024-03-24T11:35:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[37907]: pgmap v735: 174 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 168 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 715 op/s; 372/196665 objects degraded (0.189%); 8205/196665 objects misplaced (4.172%); 5.3 MiB/s, 84 objects/s recovering 2024-03-24T11:35:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-24T11:35:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[37907]: osdmap e336: 8 total, 8 up, 8 in 2024-03-24T11:35:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:33.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:34.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:34.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:33 smithi181 ceph-mon[33998]: pgmap v735: 174 pgs: 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 168 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 715 op/s; 372/196665 objects degraded (0.189%); 8205/196665 objects misplaced (4.172%); 5.3 MiB/s, 84 objects/s recovering 2024-03-24T11:35:34.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-24T11:35:34.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:33 smithi181 ceph-mon[33998]: osdmap e336: 8 total, 8 up, 8 in 2024-03-24T11:35:34.018 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:34.105 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.147 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:34.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:34.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:34.289 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:34.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:34.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:35:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:35:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:34.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:34.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.557 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:34.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:34.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:34.699 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:34.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:34.829 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:34.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:34 smithi179 ceph-mon[29608]: osdmap e337: 8 total, 8 up, 8 in 2024-03-24T11:35:34.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:34 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/196665 objects degraded (0.189%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:34.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:34 smithi179 ceph-mon[37907]: osdmap e337: 8 total, 8 up, 8 in 2024-03-24T11:35:34.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:34 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/196665 objects degraded (0.189%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:34.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:34.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:35.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:35.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:35.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:34 smithi181 ceph-mon[33998]: osdmap e337: 8 total, 8 up, 8 in 2024-03-24T11:35:35.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:34 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/196665 objects degraded (0.189%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:35.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:35.227 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:35.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:35.314 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:35:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:35:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:35:35.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:35.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:35.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:35.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:35.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:35.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:35.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:35.586 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:35.635 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:35.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:35.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:35.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:35.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:35.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:35.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:35.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:35 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:35 smithi179 ceph-mon[29608]: pgmap v738: 174 pgs: 1 active+remapped+backfill_toofull, 2 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 372/183624 objects degraded (0.203%); 6296/183624 objects misplaced (3.429%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:35:35.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:35 smithi179 ceph-mon[29608]: osdmap e338: 8 total, 8 up, 8 in 2024-03-24T11:35:35.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:35 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:35.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:35 smithi179 ceph-mon[37907]: pgmap v738: 174 pgs: 1 active+remapped+backfill_toofull, 2 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 372/183624 objects degraded (0.203%); 6296/183624 objects misplaced (3.429%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:35:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:35 smithi179 ceph-mon[37907]: osdmap e338: 8 total, 8 up, 8 in 2024-03-24T11:35:36.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:36.048 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:36.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:36.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:36.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:35 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:36.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:35 smithi181 ceph-mon[33998]: pgmap v738: 174 pgs: 1 active+remapped+backfill_toofull, 2 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 372/183624 objects degraded (0.203%); 6296/183624 objects misplaced (3.429%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:35:36.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:35 smithi181 ceph-mon[33998]: osdmap e338: 8 total, 8 up, 8 in 2024-03-24T11:35:36.182 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:36.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:36.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:36.324 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:36.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:36.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:36.465 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:36.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:36.566 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:36.609 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:36.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:36.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:36.742 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:36.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:36.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:36.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:36.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:36 smithi179 ceph-mon[29608]: osdmap e339: 8 total, 8 up, 8 in 2024-03-24T11:35:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:36 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T11:35:36.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:36 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T11:35:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:36 smithi179 ceph-mon[37907]: osdmap e339: 8 total, 8 up, 8 in 2024-03-24T11:35:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:36 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T11:35:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:36 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T11:35:36.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:37.021 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:37.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:37.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:36 smithi181 ceph-mon[33998]: osdmap e339: 8 total, 8 up, 8 in 2024-03-24T11:35:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:36 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T11:35:37.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:36 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-24T11:35:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:37.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:37.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:37.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:37.434 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:37.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.579 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:37.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:37.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:37.715 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:37.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:37.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:37.947 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:37.980 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:37.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:38.082 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:38.115 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:38.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:38.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:37 smithi181 ceph-mon[33998]: pgmap v741: 173 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 372/183267 objects degraded (0.203%); 6296/183267 objects misplaced (3.435%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:35:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:37 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-24T11:35:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:37 smithi181 ceph-mon[33998]: osdmap e340: 8 total, 8 up, 8 in 2024-03-24T11:35:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:37 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:35:38.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:37 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:35:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[29608]: pgmap v741: 173 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 372/183267 objects degraded (0.203%); 6296/183267 objects misplaced (3.435%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:35:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-24T11:35:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[29608]: osdmap e340: 8 total, 8 up, 8 in 2024-03-24T11:35:38.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:35:38.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:35:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[37907]: pgmap v741: 173 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 372/183267 objects degraded (0.203%); 6296/183267 objects misplaced (3.435%); 12 MiB/s, 185 objects/s recovering 2024-03-24T11:35:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-24T11:35:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[37907]: osdmap e340: 8 total, 8 up, 8 in 2024-03-24T11:35:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:35:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:37 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:35:38.217 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:38.250 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:38.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:38.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:38.383 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:38.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:38.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:38.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:38.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:38.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:38.645 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:38.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:38.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:38.785 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:38.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:38.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:38.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:38.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:39.035 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[29608]: 1.e scrub starts 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[29608]: 1.e scrub ok 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[29608]: osdmap e341: 8 total, 8 up, 8 in 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[29608]: 10.9 scrub starts 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[37907]: 1.e scrub starts 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[37907]: 1.e scrub ok 2024-03-24T11:35:39.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[37907]: osdmap e341: 8 total, 8 up, 8 in 2024-03-24T11:35:39.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:35:39.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:35:39.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:38 smithi179 ceph-mon[37907]: 10.9 scrub starts 2024-03-24T11:35:39.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:39.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:39.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:38 smithi181 ceph-mon[33998]: 1.e scrub starts 2024-03-24T11:35:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:38 smithi181 ceph-mon[33998]: 1.e scrub ok 2024-03-24T11:35:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:38 smithi181 ceph-mon[33998]: osdmap e341: 8 total, 8 up, 8 in 2024-03-24T11:35:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:38 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:35:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:38 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:35:39.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:38 smithi181 ceph-mon[33998]: 10.9 scrub starts 2024-03-24T11:35:39.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:39.226 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:39.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:39.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:39.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:39.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:39.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:39.503 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:39.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:39.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:39.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:39.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:39.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:39.795 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:39.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:39.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:39.934 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:39.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:40.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:40.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:40.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:40.074 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:40.078 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[29608]: pgmap v744: 173 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s; 372/177906 objects degraded (0.209%); 5060/177906 objects misplaced (2.844%); 7.8 MiB/s, 124 objects/s recovering 2024-03-24T11:35:40.078 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[29608]: 10.0 scrub starts 2024-03-24T11:35:40.078 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[29608]: 10.9 scrub ok 2024-03-24T11:35:40.078 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[29608]: 10.0 scrub ok 2024-03-24T11:35:40.078 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[29608]: osdmap e342: 8 total, 8 up, 8 in 2024-03-24T11:35:40.078 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/177906 objects degraded (0.209%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[29608]: 10.6 scrub starts 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[37907]: pgmap v744: 173 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s; 372/177906 objects degraded (0.209%); 5060/177906 objects misplaced (2.844%); 7.8 MiB/s, 124 objects/s recovering 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[37907]: 10.0 scrub starts 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[37907]: 10.9 scrub ok 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[37907]: 10.0 scrub ok 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[37907]: osdmap e342: 8 total, 8 up, 8 in 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/177906 objects degraded (0.209%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:40.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:39 smithi179 ceph-mon[37907]: 10.6 scrub starts 2024-03-24T11:35:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:39 smithi181 ceph-mon[33998]: pgmap v744: 173 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+recovering+undersized+remapped, 166 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s; 372/177906 objects degraded (0.209%); 5060/177906 objects misplaced (2.844%); 7.8 MiB/s, 124 objects/s recovering 2024-03-24T11:35:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:39 smithi181 ceph-mon[33998]: 10.0 scrub starts 2024-03-24T11:35:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:39 smithi181 ceph-mon[33998]: 10.9 scrub ok 2024-03-24T11:35:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:39 smithi181 ceph-mon[33998]: 10.0 scrub ok 2024-03-24T11:35:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:39 smithi181 ceph-mon[33998]: osdmap e342: 8 total, 8 up, 8 in 2024-03-24T11:35:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:39 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/177906 objects degraded (0.209%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:40.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:39 smithi181 ceph-mon[33998]: 10.6 scrub starts 2024-03-24T11:35:40.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:40.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:40.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:40.301 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:40.342 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:40.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:40.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:40.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:40.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:40.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:40.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:40.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:40.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:40.747 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:40.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:40.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:40.892 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:40.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[29608]: 10.7 scrub starts 2024-03-24T11:35:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[29608]: osdmap e343: 8 total, 8 up, 8 in 2024-03-24T11:35:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[29608]: 10.6 scrub ok 2024-03-24T11:35:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[29608]: 10.7 scrub ok 2024-03-24T11:35:40.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T11:35:40.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T11:35:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[37907]: 10.7 scrub starts 2024-03-24T11:35:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[37907]: osdmap e343: 8 total, 8 up, 8 in 2024-03-24T11:35:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[37907]: 10.6 scrub ok 2024-03-24T11:35:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[37907]: 10.7 scrub ok 2024-03-24T11:35:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T11:35:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:40 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T11:35:40.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:41.034 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:41.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:40 smithi181 ceph-mon[33998]: 10.7 scrub starts 2024-03-24T11:35:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:40 smithi181 ceph-mon[33998]: osdmap e343: 8 total, 8 up, 8 in 2024-03-24T11:35:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:40 smithi181 ceph-mon[33998]: 10.6 scrub ok 2024-03-24T11:35:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:40 smithi181 ceph-mon[33998]: 10.7 scrub ok 2024-03-24T11:35:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:40 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T11:35:41.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:40 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-24T11:35:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.174 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:41.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:41.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:41.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:41.446 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:41.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:41.582 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:41.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:41.724 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:41.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.852 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:41.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:41.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:41.995 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:41.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:42.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:42.138 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:42.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:42.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:41 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:35:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:41 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:35:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:41 smithi181 ceph-mon[33998]: pgmap v747: 172 pgs: 1 creating+peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 1 active+recovery_wait; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 372/170403 objects degraded (0.218%); 4237/170403 objects misplaced (2.486%); 13 MiB/s, 205 objects/s recovering 2024-03-24T11:35:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:41 smithi181 ceph-mon[33998]: 10.16 scrub starts 2024-03-24T11:35:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:41 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-24T11:35:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:41 smithi181 ceph-mon[33998]: osdmap e344: 8 total, 8 up, 8 in 2024-03-24T11:35:42.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:41 smithi181 ceph-mon[33998]: 10.16 scrub ok 2024-03-24T11:35:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:35:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:35:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[29608]: pgmap v747: 172 pgs: 1 creating+peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 1 active+recovery_wait; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 372/170403 objects degraded (0.218%); 4237/170403 objects misplaced (2.486%); 13 MiB/s, 205 objects/s recovering 2024-03-24T11:35:42.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[29608]: 10.16 scrub starts 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[29608]: osdmap e344: 8 total, 8 up, 8 in 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[29608]: 10.16 scrub ok 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[37907]: pgmap v747: 172 pgs: 1 creating+peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 1 active+recovery_wait; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 372/170403 objects degraded (0.218%); 4237/170403 objects misplaced (2.486%); 13 MiB/s, 205 objects/s recovering 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[37907]: 10.16 scrub starts 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[37907]: osdmap e344: 8 total, 8 up, 8 in 2024-03-24T11:35:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:41 smithi179 ceph-mon[37907]: 10.16 scrub ok 2024-03-24T11:35:42.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:42.281 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:42.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:42.382 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:42.423 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:42.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:42.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:42.556 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:42.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:42.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:42.693 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:42.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:42.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:42.841 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:42.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:42.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:42.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:42.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:43.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:43.104 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:42 smithi181 ceph-mon[33998]: 10.12 scrub starts 2024-03-24T11:35:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:42 smithi181 ceph-mon[33998]: osdmap e345: 8 total, 8 up, 8 in 2024-03-24T11:35:43.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:42 smithi181 ceph-mon[33998]: 10.12 scrub ok 2024-03-24T11:35:43.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:43.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:42 smithi179 ceph-mon[29608]: 10.12 scrub starts 2024-03-24T11:35:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:42 smithi179 ceph-mon[29608]: osdmap e345: 8 total, 8 up, 8 in 2024-03-24T11:35:43.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:42 smithi179 ceph-mon[29608]: 10.12 scrub ok 2024-03-24T11:35:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:42 smithi179 ceph-mon[37907]: 10.12 scrub starts 2024-03-24T11:35:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:42 smithi179 ceph-mon[37907]: osdmap e345: 8 total, 8 up, 8 in 2024-03-24T11:35:43.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:42 smithi179 ceph-mon[37907]: 10.12 scrub ok 2024-03-24T11:35:43.244 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:43.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.389 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:43.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:43.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:43.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:43.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:43.652 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:43.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:43.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:43.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:43.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:43.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:44.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:44.047 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:44.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:44.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:43 smithi181 ceph-mon[33998]: pgmap v750: 172 pgs: 1 creating+peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 1 active+recovery_wait; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 372/170403 objects degraded (0.218%); 4237/170403 objects misplaced (2.486%); 5.0 MiB/s, 80 objects/s recovering 2024-03-24T11:35:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:43 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:35:44.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:43 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:35:44.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:43 smithi181 ceph-mon[33998]: osdmap e346: 8 total, 8 up, 8 in 2024-03-24T11:35:44.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:35:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:35:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:44.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:44.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[37907]: pgmap v750: 172 pgs: 1 creating+peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 1 active+recovery_wait; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 372/170403 objects degraded (0.218%); 4237/170403 objects misplaced (2.486%); 5.0 MiB/s, 80 objects/s recovering 2024-03-24T11:35:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:35:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:35:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[37907]: osdmap e346: 8 total, 8 up, 8 in 2024-03-24T11:35:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[29608]: pgmap v750: 172 pgs: 1 creating+peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 1 active+recovery_wait; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 372/170403 objects degraded (0.218%); 4237/170403 objects misplaced (2.486%); 5.0 MiB/s, 80 objects/s recovering 2024-03-24T11:35:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:35:44.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:35:44.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:43 smithi179 ceph-mon[29608]: osdmap e346: 8 total, 8 up, 8 in 2024-03-24T11:35:44.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:44.320 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:44.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:44.422 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:44.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:44.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:44.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:44.606 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:44.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:44.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:44.746 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:44.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:44.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:44.850 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[37907]: osdmap e347: 8 total, 8 up, 8 in 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/170403 objects degraded (0.218%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[37907]: osdmap e348: 8 total, 8 up, 8 in 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[37907]: 10.11 scrub starts 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[29608]: osdmap e347: 8 total, 8 up, 8 in 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/170403 objects degraded (0.218%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[29608]: osdmap e348: 8 total, 8 up, 8 in 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[29608]: 10.11 scrub starts 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T11:35:44.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:44 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T11:35:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:44.880 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:44.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:44.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:45.009 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:45.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.135 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:45.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:45.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:44 smithi181 ceph-mon[33998]: osdmap e347: 8 total, 8 up, 8 in 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:44 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/170403 objects degraded (0.218%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:44 smithi181 ceph-mon[33998]: osdmap e348: 8 total, 8 up, 8 in 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:44 smithi181 ceph-mon[33998]: 10.11 scrub starts 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:44 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T11:35:45.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:44 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-24T11:35:45.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:35:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:35:45.236 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:45.262 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:45.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:45.396 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:45.498 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.539 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:45.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:45.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:45.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:45.771 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:45.806 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:45.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:45.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:45.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[29608]: pgmap v754: 171 pgs: 2 peering, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 372/157020 objects degraded (0.237%); 1880/157020 objects misplaced (1.197%) 2024-03-24T11:35:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[29608]: 10.11 scrub ok 2024-03-24T11:35:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[29608]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:35:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-24T11:35:45.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[29608]: osdmap e349: 8 total, 8 up, 8 in 2024-03-24T11:35:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[37907]: pgmap v754: 171 pgs: 2 peering, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 372/157020 objects degraded (0.237%); 1880/157020 objects misplaced (1.197%) 2024-03-24T11:35:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[37907]: 10.11 scrub ok 2024-03-24T11:35:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[37907]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:35:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-24T11:35:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:45 smithi179 ceph-mon[37907]: osdmap e349: 8 total, 8 up, 8 in 2024-03-24T11:35:46.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:46.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:46.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:46.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:45 smithi181 ceph-mon[33998]: pgmap v754: 171 pgs: 2 peering, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 372/157020 objects degraded (0.237%); 1880/157020 objects misplaced (1.197%) 2024-03-24T11:35:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:45 smithi181 ceph-mon[33998]: 10.11 scrub ok 2024-03-24T11:35:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:45 smithi181 ceph-mon[33998]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:35:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:45 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-24T11:35:46.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:45 smithi181 ceph-mon[33998]: osdmap e349: 8 total, 8 up, 8 in 2024-03-24T11:35:46.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:46.184 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:46.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:46.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:46.389 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:35:46.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:46.452 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:46.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:46.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:46.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.727 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:46.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:46.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:46.862 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:46.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:46.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:46.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:47.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:47.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:47.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:47.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/976445870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:35:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:46 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:35:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:46 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:35:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:46 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:35:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:46 smithi181 ceph-mon[33998]: osdmap e350: 8 total, 8 up, 8 in 2024-03-24T11:35:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:46 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:35:47.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:46 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:35:47.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/976445870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:35:47.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:35:47.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[37907]: osdmap e350: 8 total, 8 up, 8 in 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/976445870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[29608]: osdmap e350: 8 total, 8 up, 8 in 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:35:47.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:46 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:35:47.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:47.262 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:47.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:47.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:47.402 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:47.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:47.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:47.535 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:47.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:47.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:47.675 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:47.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:47.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:47.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:47.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:47.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:47.960 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:47.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:48.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:48.096 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:48.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:47 smithi181 ceph-mon[33998]: pgmap v757: 171 pgs: 2 peering, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 372/156588 objects degraded (0.238%); 1880/156588 objects misplaced (1.201%) 2024-03-24T11:35:48.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:47 smithi181 ceph-mon[33998]: osdmap e351: 8 total, 8 up, 8 in 2024-03-24T11:35:48.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:47 smithi179 ceph-mon[29608]: pgmap v757: 171 pgs: 2 peering, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 372/156588 objects degraded (0.238%); 1880/156588 objects misplaced (1.201%) 2024-03-24T11:35:48.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:47 smithi179 ceph-mon[29608]: osdmap e351: 8 total, 8 up, 8 in 2024-03-24T11:35:48.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:47 smithi179 ceph-mon[37907]: pgmap v757: 171 pgs: 2 peering, 2 active+recovering+undersized+remapped, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 372/156588 objects degraded (0.238%); 1880/156588 objects misplaced (1.201%) 2024-03-24T11:35:48.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:47 smithi179 ceph-mon[37907]: osdmap e351: 8 total, 8 up, 8 in 2024-03-24T11:35:48.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:48.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:48.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:48.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:48.356 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:48.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:48.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:48.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:48.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:48.599 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:48.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:48.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:48.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:48.781 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:48.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:48.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:48.913 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:48.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:49.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:49.048 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:49.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:49.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:49.184 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:49.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:49.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:49.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:49.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:49.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:49.456 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:49.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:49.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:49.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:49.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:49.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: osdmap e352: 8 total, 8 up, 8 in 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: pgmap v760: 170 pgs: 1 activating, 2 peering, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 371 op/s; 372/152571 objects degraded (0.244%); 1368/152571 objects misplaced (0.897%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/156588 objects degraded (0.238%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-24T11:35:49.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:49 smithi181 ceph-mon[33998]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T11:35:49.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:35:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: osdmap e352: 8 total, 8 up, 8 in 2024-03-24T11:35:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: pgmap v760: 170 pgs: 1 activating, 2 peering, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 371 op/s; 372/152571 objects degraded (0.244%); 1368/152571 objects misplaced (0.897%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T11:35:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T11:35:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T11:35:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:35:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/156588 objects degraded (0.238%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[29608]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: osdmap e352: 8 total, 8 up, 8 in 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: pgmap v760: 170 pgs: 1 activating, 2 peering, 165 active+clean, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 371 op/s; 372/152571 objects degraded (0.244%); 1368/152571 objects misplaced (0.897%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:35:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:35:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/156588 objects degraded (0.238%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-24T11:35:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-24T11:35:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:49 smithi179 ceph-mon[37907]: osdmap e353: 8 total, 8 up, 8 in 2024-03-24T11:35:49.723 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:49.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:49.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:49.871 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:49.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:49.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:50.011 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:50.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:50.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:50.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:50.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:50.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:50.443 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:50.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.575 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:50.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:50 smithi181 ceph-mon[33998]: 10.15 deep-scrub starts 2024-03-24T11:35:50.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:50 smithi181 ceph-mon[33998]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T11:35:50.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:50.707 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:50 smithi179 ceph-mon[29608]: 10.15 deep-scrub starts 2024-03-24T11:35:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:50 smithi179 ceph-mon[29608]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T11:35:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:50 smithi179 ceph-mon[37907]: 10.15 deep-scrub starts 2024-03-24T11:35:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:50 smithi179 ceph-mon[37907]: osdmap e354: 8 total, 8 up, 8 in 2024-03-24T11:35:50.809 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:50.846 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:50.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:50.990 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:50.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:51.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:51.126 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:51.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:51.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:51.278 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:51.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:51.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:51.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:51.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:51.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:51.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:51.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:51.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:51.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:51.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:51 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:35:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:51 smithi181 ceph-mon[33998]: 6.1f scrub ok 2024-03-24T11:35:51.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:51 smithi181 ceph-mon[33998]: pgmap v763: 170 pgs: 2 peering, 1 activating, 165 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 372/143376 objects degraded (0.259%); 662/143376 objects misplaced (0.462%) 2024-03-24T11:35:51.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:51 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:35:51.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:51 smithi179 ceph-mon[29608]: 6.1f scrub ok 2024-03-24T11:35:51.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:51 smithi179 ceph-mon[29608]: pgmap v763: 170 pgs: 2 peering, 1 activating, 165 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 372/143376 objects degraded (0.259%); 662/143376 objects misplaced (0.462%) 2024-03-24T11:35:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:51 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:35:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:51 smithi179 ceph-mon[37907]: 6.1f scrub ok 2024-03-24T11:35:51.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:51 smithi179 ceph-mon[37907]: pgmap v763: 170 pgs: 2 peering, 1 activating, 165 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 372/143376 objects degraded (0.259%); 662/143376 objects misplaced (0.462%) 2024-03-24T11:35:51.683 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:51.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:51.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:51.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:51.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:51.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:51.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:51.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:52.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:52.091 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:52.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:52.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:52.232 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:52.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:52.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:52.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:52.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:52.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:52.517 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:52.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:52.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:52.636 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:52 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:35:52.636 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:52 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:35:52.636 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:52 smithi179 ceph-mon[29608]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T11:35:52.636 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:52 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:35:52.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:52 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:35:52.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:52 smithi179 ceph-mon[37907]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T11:35:52.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:52 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:35:52.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:52 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:35:52.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:52 smithi181 ceph-mon[33998]: osdmap e355: 8 total, 8 up, 8 in 2024-03-24T11:35:52.659 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:52.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:52.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:52.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:52.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:52.887 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:35:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "success": "" 2024-03-24T11:35:52.888 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:35:52.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:52.926 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:52.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:52.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:52.955 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:35:52.955 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:35:53.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:53.059 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:53.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:53.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:53.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:53.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:53.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:53.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:53.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:53.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:53.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:53.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:53.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:53.598 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: 3.1e scrub starts 2024-03-24T11:35:53.598 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: 3.1e scrub ok 2024-03-24T11:35:53.598 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:35:53.598 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:35:53.598 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: pgmap v766: 169 pgs: 1 peering, 1 activating, 165 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 372/143373 objects degraded (0.259%); 662/143373 objects misplaced (0.462%) 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: 3.1e scrub starts 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: 3.1e scrub ok 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: pgmap v766: 169 pgs: 1 peering, 1 activating, 165 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 372/143373 objects degraded (0.259%); 662/143373 objects misplaced (0.462%) 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:53.599 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:53 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:53.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:53.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: 3.1e scrub starts 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: 3.1e scrub ok 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: osdmap e356: 8 total, 8 up, 8 in 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: pgmap v766: 169 pgs: 1 peering, 1 activating, 165 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 372/143373 objects degraded (0.259%); 662/143373 objects misplaced (0.462%) 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:53.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:53 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T11:35:53.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:53.755 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:53.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:53.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:53.896 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:53.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:53.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:54.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:54.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:54.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:35:54.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:54.160 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:54.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:54.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:54.298 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:54.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:54.374 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:35:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:35:54.400 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:54.454 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:54.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:54.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:54.592 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:54.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:54.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:35:54.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[37907]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T11:35:54.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[37907]: 5.9 deep-scrub starts 2024-03-24T11:35:54.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[37907]: 5.9 deep-scrub ok 2024-03-24T11:35:54.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[37907]: Health check update: Degraded data redundancy: 372/143373 objects degraded (0.259%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-24T11:35:54.634 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:35:54.634 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[29608]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T11:35:54.634 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[29608]: 5.9 deep-scrub starts 2024-03-24T11:35:54.634 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[29608]: 5.9 deep-scrub ok 2024-03-24T11:35:54.634 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:54 smithi179 ceph-mon[29608]: Health check update: Degraded data redundancy: 372/143373 objects degraded (0.259%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-24T11:35:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:54 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T11:35:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:54 smithi181 ceph-mon[33998]: osdmap e357: 8 total, 8 up, 8 in 2024-03-24T11:35:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:54 smithi181 ceph-mon[33998]: 5.9 deep-scrub starts 2024-03-24T11:35:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:54 smithi181 ceph-mon[33998]: 5.9 deep-scrub ok 2024-03-24T11:35:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:54 smithi181 ceph-mon[33998]: Health check update: Degraded data redundancy: 372/143373 objects degraded (0.259%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-24T11:35:54.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:54.722 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:54.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:54.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:54.861 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:54.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:54.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:55.006 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:55.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:55.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:55.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:55.268 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:55.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:55.385 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:35:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:35:55.411 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:55.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:55.546 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:55.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:55.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:55 smithi181 ceph-mon[33998]: 4.17 scrub starts 2024-03-24T11:35:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:55 smithi181 ceph-mon[33998]: 4.17 scrub ok 2024-03-24T11:35:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:55 smithi181 ceph-mon[33998]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T11:35:55.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:55 smithi181 ceph-mon[33998]: pgmap v769: 169 pgs: 1 peering, 166 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 240/129024 objects degraded (0.186%); 240/129024 objects misplaced (0.186%); 2.2 MiB/s, 68 objects/s recovering 2024-03-24T11:35:55.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:55.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[29608]: 4.17 scrub starts 2024-03-24T11:35:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[29608]: 4.17 scrub ok 2024-03-24T11:35:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[29608]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T11:35:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[29608]: pgmap v769: 169 pgs: 1 peering, 166 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 240/129024 objects degraded (0.186%); 240/129024 objects misplaced (0.186%); 2.2 MiB/s, 68 objects/s recovering 2024-03-24T11:35:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[37907]: 4.17 scrub starts 2024-03-24T11:35:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[37907]: 4.17 scrub ok 2024-03-24T11:35:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:35:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[37907]: osdmap e358: 8 total, 8 up, 8 in 2024-03-24T11:35:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:55 smithi179 ceph-mon[37907]: pgmap v769: 169 pgs: 1 peering, 166 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 240/129024 objects degraded (0.186%); 240/129024 objects misplaced (0.186%); 2.2 MiB/s, 68 objects/s recovering 2024-03-24T11:35:55.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:55.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:55.853 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:55.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:55.975 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:55.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:56.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:56.121 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:56.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:56.224 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:56.261 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:56.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:56.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:56.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:56.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:56.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:56.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:56.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:56.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:56.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:35:56.688 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:56.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:56 smithi179 ceph-mon[29608]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T11:35:56.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:56 smithi179 ceph-mon[37907]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T11:35:56.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:56.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:56.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:56 smithi181 ceph-mon[33998]: osdmap e359: 8 total, 8 up, 8 in 2024-03-24T11:35:56.924 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:56.996 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:56.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:57.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:57.147 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:57.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:57.249 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:57.300 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:57.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:57.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:57.439 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:57.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:57.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:57.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:57.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:57.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: 5.1f deep-scrub starts 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: 5.1f deep-scrub ok 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: pgmap v771: 168 pgs: 166 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 240/128604 objects degraded (0.187%); 240/128604 objects misplaced (0.187%); 2.3 MiB/s, 68 objects/s recovering 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: 5.1f deep-scrub starts 2024-03-24T11:35:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: 5.1f deep-scrub ok 2024-03-24T11:35:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: pgmap v771: 168 pgs: 166 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 240/128604 objects degraded (0.187%); 240/128604 objects misplaced (0.187%); 2.3 MiB/s, 68 objects/s recovering 2024-03-24T11:35:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T11:35:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:35:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:35:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:35:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:57 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:35:57.714 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:57.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:57.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:57.841 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:57.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: 5.1f deep-scrub starts 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: 5.1f deep-scrub ok 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: pgmap v771: 168 pgs: 166 active+clean, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 240/128604 objects degraded (0.187%); 240/128604 objects misplaced (0.187%); 2.3 MiB/s, 68 objects/s recovering 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: osdmap e360: 8 total, 8 up, 8 in 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:35:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:57 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:35:57.930 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:35:57.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:35:57.957 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:35:57.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:57.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:35:57.978 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:35:58.005 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:35:58.030 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:35:58.050 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:35:58.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:58.070 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:35:58.090 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:35:58.091 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:35:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:58.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:58.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:58.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:58.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:58.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:58.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:35:58.343 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:58.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:35:58.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:35:58.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:58.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:35:58.552 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:35:58.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:58.617 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:58.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:58.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 10.2 scrub starts 2024-03-24T11:35:58.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 10.15 deep-scrub ok 2024-03-24T11:35:58.632 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 10.2 scrub ok 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3700312768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 10.2 scrub starts 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 10.15 deep-scrub ok 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 10.2 scrub ok 2024-03-24T11:35:58.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:35:58.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:35:58.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T11:35:58.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:35:58.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:35:58.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:35:58.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:35:58.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3700312768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:58.718 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:58.758 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:58.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:58.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:58.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 10.2 scrub starts 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 10.15 deep-scrub ok 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 10.2 scrub ok 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: osdmap e361: 8 total, 8 up, 8 in 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:35:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3700312768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:35:58.907 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:58.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:59.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:59.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:59.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:59.142 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:35:59.177 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:59.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:35:59.215 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:35:59.215 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:35:59.215 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:35:59.215 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:35:59.215 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:35:59.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:59.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:59.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:59.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:35:59.446 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:59.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:35:59.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:35:59.588 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:59.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:35:59.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 10.13 scrub starts 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 10.13 scrub ok 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: pgmap v774: 168 pgs: 1 creating+peering, 166 active+clean, 1 active+clean+scrubbing+deep; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 240/128604 objects degraded (0.187%), 1 pg degraded, 1 pg undersized) 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:35:59.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 5.d deep-scrub starts 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 5.d deep-scrub ok 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 10.8 scrub starts 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: 10.8 scrub ok 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[29608]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 10.13 scrub starts 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 10.13 scrub ok 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:35:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: pgmap v774: 168 pgs: 1 creating+peering, 166 active+clean, 1 active+clean+scrubbing+deep; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 240/128604 objects degraded (0.187%), 1 pg degraded, 1 pg undersized) 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 5.d deep-scrub starts 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 5.d deep-scrub ok 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 10.8 scrub starts 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: 10.8 scrub ok 2024-03-24T11:35:59.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:35:59 smithi179 ceph-mon[37907]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T11:35:59.728 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:35:59.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:59.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:35:59.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:35:59.859 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:35:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:35:59.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 10.13 scrub starts 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 10.13 scrub ok 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: pgmap v774: 168 pgs: 1 creating+peering, 166 active+clean, 1 active+clean+scrubbing+deep; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 240/128604 objects degraded (0.187%), 1 pg degraded, 1 pg undersized) 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: osdmap e362: 8 total, 8 up, 8 in 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 5.d deep-scrub starts 2024-03-24T11:35:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 5.d deep-scrub ok 2024-03-24T11:35:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 10.8 scrub starts 2024-03-24T11:35:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:35:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:35:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: 10.8 scrub ok 2024-03-24T11:35:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:35:59 smithi181 ceph-mon[33998]: osdmap e363: 8 total, 8 up, 8 in 2024-03-24T11:35:59.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:35:59.999 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:00.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:00.093 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:36:00.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:36:00.093 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-24T11:36:00.093 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:36:00.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:00.181 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:00.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:00.284 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:00.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:00.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:00.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:00.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:00.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:00.503 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:36:00.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:00.639 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:00.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[29608]: 10.19 scrub starts 2024-03-24T11:36:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[29608]: 10.19 scrub ok 2024-03-24T11:36:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[29608]: 10.18 deep-scrub starts 2024-03-24T11:36:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:36:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:36:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[29608]: 10.4 scrub starts 2024-03-24T11:36:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[29608]: 10.4 scrub ok 2024-03-24T11:36:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[37907]: 10.19 scrub starts 2024-03-24T11:36:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[37907]: 10.19 scrub ok 2024-03-24T11:36:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[37907]: 10.18 deep-scrub starts 2024-03-24T11:36:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:36:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:36:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[37907]: 10.4 scrub starts 2024-03-24T11:36:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:00 smithi179 ceph-mon[37907]: 10.4 scrub ok 2024-03-24T11:36:00.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:00.774 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:00.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:00.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:00 smithi181 ceph-mon[33998]: 10.19 scrub starts 2024-03-24T11:36:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:00 smithi181 ceph-mon[33998]: 10.19 scrub ok 2024-03-24T11:36:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:00 smithi181 ceph-mon[33998]: 10.18 deep-scrub starts 2024-03-24T11:36:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:00 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:36:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:00 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:36:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:00 smithi181 ceph-mon[33998]: 10.4 scrub starts 2024-03-24T11:36:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:00 smithi181 ceph-mon[33998]: 10.4 scrub ok 2024-03-24T11:36:00.920 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:00.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:01.022 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:01.066 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:01.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:01.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:01.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:01.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:01.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:01.335 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:01.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:01.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:01.468 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:01.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:01.473 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:36:01.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:01.655 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:01.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:01.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:01.782 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:01.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:01.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 10.14 scrub starts 2024-03-24T11:36:01.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 10.14 scrub ok 2024-03-24T11:36:01.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 10.1 scrub starts 2024-03-24T11:36:01.802 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: pgmap v777: 168 pgs: 1 creating+peering, 167 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T11:36:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 10.1 scrub ok 2024-03-24T11:36:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:36:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:36:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 1.2 scrub starts 2024-03-24T11:36:01.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:01 smithi181 ceph-mon[33998]: 1.2 scrub ok 2024-03-24T11:36:01.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:01.912 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:01.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 10.14 scrub starts 2024-03-24T11:36:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 10.14 scrub ok 2024-03-24T11:36:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 10.1 scrub starts 2024-03-24T11:36:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: pgmap v777: 168 pgs: 1 creating+peering, 167 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 10.1 scrub ok 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 1.2 scrub starts 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[29608]: 1.2 scrub ok 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 10.14 scrub starts 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 10.14 scrub ok 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 10.1 scrub starts 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: pgmap v777: 168 pgs: 1 creating+peering, 167 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: osdmap e364: 8 total, 8 up, 8 in 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 10.1 scrub ok 2024-03-24T11:36:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:36:01.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:36:01.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 1.2 scrub starts 2024-03-24T11:36:01.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:01 smithi179 ceph-mon[37907]: 1.2 scrub ok 2024-03-24T11:36:02.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:02.043 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:02.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:02.097 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:02.145 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:02.171 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:02.224 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:02.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:02.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:02.358 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:02.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:02.460 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:02.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:02.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:02.541 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:02 smithi179 ceph-mon[29608]: 10.3 deep-scrub starts 2024-03-24T11:36:02.541 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:02 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:36:02.541 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:02 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:36:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:02 smithi179 ceph-mon[37907]: 10.3 deep-scrub starts 2024-03-24T11:36:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:02 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:36:02.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:02 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:36:02.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:02.637 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:02.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:02.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:02.784 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:02.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:02.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:02.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:02 smithi181 ceph-mon[33998]: 10.3 deep-scrub starts 2024-03-24T11:36:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:02 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:36:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:02 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:36:02.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:02.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:03.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:03.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:03.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:03.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:03.209 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:03.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:03.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:03.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:03.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:03.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:03.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:03.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:03.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:03.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:03.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:03.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:03.751 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:03.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:03.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:03.898 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:03.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 1.8 scrub starts 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 1.8 scrub ok 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 10.10 scrub starts 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: pgmap v779: 168 pgs: 1 creating+peering, 167 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 10.10 scrub ok 2024-03-24T11:36:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T11:36:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 10.5 scrub starts 2024-03-24T11:36:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 10.5 scrub ok 2024-03-24T11:36:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:36:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:03 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:36:03.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 1.8 scrub starts 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 1.8 scrub ok 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 10.10 scrub starts 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: pgmap v779: 168 pgs: 1 creating+peering, 167 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 10.10 scrub ok 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 10.5 scrub starts 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 10.5 scrub ok 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:36:03.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 1.8 scrub starts 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 1.8 scrub ok 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 10.10 scrub starts 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: pgmap v779: 168 pgs: 1 creating+peering, 167 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 10.10 scrub ok 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: osdmap e365: 8 total, 8 up, 8 in 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 10.5 scrub starts 2024-03-24T11:36:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 10.5 scrub ok 2024-03-24T11:36:03.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:36:03.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:03 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:36:04.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:04.029 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:04.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:04.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:04.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:04.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:04.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:04.295 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:04.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:04.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:04.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:36:04.451 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:04.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:04.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:04.571 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[29608]: 10.17 scrub starts 2024-03-24T11:36:04.571 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[29608]: 10.17 scrub ok 2024-03-24T11:36:04.571 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[37907]: 10.17 scrub starts 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[37907]: 10.17 scrub ok 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:36:04.572 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:04 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:36:04.596 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:04.601 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:04.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:04.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:04.735 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:04.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:04.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:04.879 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:04.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-mon[33998]: 10.17 scrub starts 2024-03-24T11:36:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-mon[33998]: 10.17 scrub ok 2024-03-24T11:36:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:36:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:36:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:04 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:36:04.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:05.043 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:05.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:05.086 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:36:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:36:05.145 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:05.225 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:05.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:05.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:05.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:05.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:05.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:05.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:05.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:05.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:05.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:05.673 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:05.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: pgmap v781: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T11:36:05.689 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:36:05.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: pgmap v781: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail 2024-03-24T11:36:05.691 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T11:36:05.691 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.691 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.691 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.691 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:05 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:05.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:05.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:36:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:36:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:36:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:36:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: pgmap v781: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail 2024-03-24T11:36:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: osdmap e366: 8 total, 8 up, 8 in 2024-03-24T11:36:05.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:05 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:05.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:05.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:05.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:06.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:06.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:06.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:06.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:06.239 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:06.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:06.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:06.365 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:06.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:06.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:06.499 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:36:06.499 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:06.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:06.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:06.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:06.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: 10.b scrub starts 2024-03-24T11:36:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3606361200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:36:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:36:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[37907]: 10.b scrub ok 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: 10.b scrub starts 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3606361200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:36:06.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:06 smithi179 ceph-mon[29608]: 10.b scrub ok 2024-03-24T11:36:06.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:06.775 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:06.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:06.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: 10.b scrub starts 2024-03-24T11:36:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3606361200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:36:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:36:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:36:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:36:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:36:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:36:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:06 smithi181 ceph-mon[33998]: 10.b scrub ok 2024-03-24T11:36:06.908 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:06.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:07.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:07.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:07.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:07.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:07.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:07.304 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:07.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:07.442 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:07.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:07.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:07.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:07.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:07.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:07.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: pgmap v783: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:07 smithi181 ceph-mon[33998]: 10.18 deep-scrub ok 2024-03-24T11:36:07.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: pgmap v783: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail 2024-03-24T11:36:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:36:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T11:36:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:07.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[37907]: 10.18 deep-scrub ok 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: pgmap v783: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 165 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: osdmap e367: 8 total, 8 up, 8 in 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:07.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:07 smithi179 ceph-mon[29608]: 10.18 deep-scrub ok 2024-03-24T11:36:07.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:07.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:08.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:08.119 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:08.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:08.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:08.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:08.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:08.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:08.408 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:08.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:08.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:08.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:08.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:08.631 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:36:08.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:08.671 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:36:08.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:08.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:08.727 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:36:08.782 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:36:08.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:08.825 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:08.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:08.833 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:36:08.876 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:36:08.919 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:36:08.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:08.942 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:186610 10.18 10:1ff375dd:::benchmark_data_smithi179_155357_object76114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:01.511006+0000", 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.5868067090000002, 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16426442099999999, 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.943 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 186610 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.511006+0000", 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.511006+0000", 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.944 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.511008+0000", 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.86e-06 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.511015+0000", 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9369999999999998e-06 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.511029+0000", 2024-03-24T11:36:08.945 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3787e-05 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.511037+0000", 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9449999999999993e-06 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.633785+0000", 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12274763700000001 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.946 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.633919+0000", 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013461300000000001 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.634343+0000", 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000423404 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.640672+0000", 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0063288700000000003 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.947 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.644653+0000", 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039812270000000004 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.675106+0000", 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.030453329000000001 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.675216+0000", 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000110073 2024-03-24T11:36:08.948 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:01.675271+0000", 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4738999999999999e-05 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:187025 10.3 10:c1efd07a:::benchmark_data_smithi179_155357_object76529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:02.021756+0000", 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0760567710000002, 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17042378699999999, 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.949 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 187025 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.021756+0000", 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.950 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.021756+0000", 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.021758+0000", 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.761e-06 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.021764+0000", 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5589999999999996e-06 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.951 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.021777+0000", 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3502e-05 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.021786+0000", 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7509999999999997e-06 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.131928+0000", 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.110141779 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.952 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.132063+0000", 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000135525 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.132545+0000", 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048189100000000003 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.146524+0000", 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013979132999999999 2024-03-24T11:36:08.953 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.146536+0000", 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1314999999999999e-05 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.191874+0000", 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.045337903999999998 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.954 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.192120+0000", 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024666899999999998 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.192180+0000", 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9997999999999997e-05 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.955 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.15621.0:187040 10.8 e364/288 10:123437a6:::benchmark_data_smithi179_155357_object76544:head v 364'8415, mlcod=364'8413)", 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:02.045584+0000", 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 4.0522288, 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15876531499999999, 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.045584+0000", 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.956 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.045584+0000", 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.045586+0000", 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.226e-06 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.045595+0000", 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0920000000000004e-06 2024-03-24T11:36:08.957 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.045613+0000", 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8128999999999999e-05 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.045623+0000", 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.9799999999999993e-06 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.958 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.187099+0000", 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14147594799999999 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.187119+0000", 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0293e-05 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.204185+0000", 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017065751000000001 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.959 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.204350+0000", 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016489599999999999 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:187378 10.18 10:1cd69434:::benchmark_data_smithi179_155357_object76882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:02.542110+0000", 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.5557029619999998, 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15396195100000001, 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.960 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 187378 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.542110+0000", 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.961 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.542110+0000", 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.542112+0000", 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.542e-06 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.542118+0000", 2024-03-24T11:36:08.962 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.9190000000000004e-06 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.542135+0000", 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7056000000000001e-05 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.542144+0000", 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9190000000000005e-06 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.963 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.643162+0000", 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.101018766 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.643368+0000", 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00020541799999999999 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.643637+0000", 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000269132 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.964 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.660962+0000", 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017325397999999999 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.666387+0000", 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0054243520000000003 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.695515+0000", 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.029128517 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.965 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.696014+0000", 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00049925100000000004 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.696072+0000", 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7680999999999999e-05 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:187542 10.3 10:c02c073d:::benchmark_data_smithi179_155357_object77046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:02.724189+0000", 2024-03-24T11:36:08.966 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.3736244979999999, 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16106582699999999, 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 187542 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.724189+0000", 2024-03-24T11:36:08.967 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.724189+0000", 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.724190+0000", 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.637e-06 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.968 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.724195+0000", 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.118e-06 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.724209+0000", 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.31e-05 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.724217+0000", 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1200000000000002e-06 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.969 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.848893+0000", 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.124676843 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.849030+0000", 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013662900000000001 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.849534+0000", 2024-03-24T11:36:08.970 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00050365200000000005 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.863278+0000", 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013744562 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.863291+0000", 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2864000000000001e-05 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.971 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.885060+0000", 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021768661000000002 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.885183+0000", 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012309199999999999 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.885255+0000", 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1549000000000002e-05 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.972 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:187544 10.3 10:c4666132:::benchmark_data_smithi179_155357_object77048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:02.725892+0000", 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.3719209920000002, 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.26750059700000001, 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 187544 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.973 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.725892+0000", 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.725892+0000", 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.974 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.725893+0000", 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.159e-06 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.725897+0000", 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8800000000000001e-06 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.725912+0000", 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5160999999999999e-05 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.975 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.725921+0000", 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8780000000000004e-06 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.852232+0000", 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.12631041700000001 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.852339+0000", 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107631 2024-03-24T11:36:08.976 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.852833+0000", 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00049348600000000001 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.863297+0000", 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.010463821999999999 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.863445+0000", 2024-03-24T11:36:08.977 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014862100000000001 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.993270+0000", 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.129824627 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.993331+0000", 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1447999999999998e-05 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.978 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.993393+0000", 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1467e-05 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:187547 10.3 10:c1ae8150:::benchmark_data_smithi179_155357_object77051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:02.728284+0000", 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.3695293890000002, 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.26559196499999999, 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.979 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 187547 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.728284+0000", 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.728284+0000", 2024-03-24T11:36:08.980 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.728285+0000", 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4389999999999999e-06 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.728291+0000", 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6860000000000003e-06 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.981 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.728310+0000", 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9213999999999999e-05 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.728319+0000", 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4090000000000003e-06 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.859926+0000", 2024-03-24T11:36:08.982 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13160752000000001 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.860043+0000", 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011650300000000001 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.860563+0000", 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00052017399999999998 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.983 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.870011+0000", 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0094484559999999992 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.870048+0000", 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.7141000000000002e-05 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.993606+0000", 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.123558031 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.984 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.993820+0000", 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000213222 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.993876+0000", 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6169999999999999e-05 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.985 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:187565 10.3 10:c4e7b4e1:::benchmark_data_smithi179_155357_object77069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:02.749676+0000", 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 3.3481372189999998, 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.24440520700000001, 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 187565 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.986 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.749676+0000", 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.749676+0000", 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.749678+0000", 2024-03-24T11:36:08.987 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6330000000000001e-06 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.749684+0000", 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6409999999999996e-06 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.749698+0000", 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3988999999999999e-05 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.988 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.749707+0000", 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4670000000000004e-06 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.863589+0000", 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11388240099999999 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.863709+0000", 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000120287 2024-03-24T11:36:08.989 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.864681+0000", 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00097114100000000002 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.891555+0000", 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.026874746000000001 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.990 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.895276+0000", 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0037208919999999999 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.993981+0000", 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.098704807000000006 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.994030+0000", 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9030000000000003e-05 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.991 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:02.994081+0000", 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1172999999999999e-05 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:188298 10.3 10:c131754a:::benchmark_data_smithi179_155357_object77802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:03.641594+0000", 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.4562195149999999, 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.19343537599999999, 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.992 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188298 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.641594+0000", 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.993 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.641594+0000", 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.641595+0000", 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4759999999999999e-06 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.641600+0000", 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4159999999999997e-06 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.994 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.641609+0000", 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.7329999999999998e-06 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.641617+0000", 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4850000000000003e-06 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.663507+0000", 2024-03-24T11:36:08.995 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021890169000000001 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.663631+0000", 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000123683 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.664123+0000", 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00049195900000000004 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.996 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.669438+0000", 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0053156050000000002 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.688514+0000", 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019075569000000001 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.834868+0000", 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.14635432000000001 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.997 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.834946+0000", 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7767000000000003e-05 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.835029+0000", 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3194000000000003e-05 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.998 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:188324 10.3 10:c538319c:::benchmark_data_smithi179_155357_object77828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:03.674982+0000", 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.422831483, 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16024295599999999, 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188324 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:08.999 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.674982+0000", 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.674982+0000", 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.674982+0000", 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.13e-07 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.000 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.674985+0000", 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.3580000000000001e-06 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.674991+0000", 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1120000000000001e-06 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.001 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.674997+0000", 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2709999999999998e-06 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.688926+0000", 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.013928901 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.689039+0000", 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112842 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.689580+0000", 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00054109500000000001 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.695449+0000", 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005869567 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.699418+0000", 2024-03-24T11:36:09.003 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0039684880000000001 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.835115+0000", 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13569700400000001 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.835165+0000", 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9734999999999997e-05 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.004 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.835225+0000", 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0069999999999999e-05 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:188408 10.18 10:185a2b21:::benchmark_data_smithi179_155357_object77912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:03.772113+0000", 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 2.325700162, 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.155728176, 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.005 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188408 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.772113+0000", 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.006 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.772113+0000", 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.772114+0000", 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3689999999999999e-06 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.772118+0000", 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.783e-06 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.007 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.772127+0000", 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7909999999999993e-06 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.772134+0000", 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.567e-06 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.790108+0000", 2024-03-24T11:36:09.008 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.017974549999999999 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.790256+0000", 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014813900000000001 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.790718+0000", 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00046181100000000001 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.009 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.907137+0000", 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.116418619 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.907181+0000", 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.4264000000000002e-05 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.010 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.927723+0000", 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.020542464999999999 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.927774+0000", 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0800999999999997e-05 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:03.927841+0000", 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7016999999999999e-05 2024-03-24T11:36:09.011 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.012 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.013 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.013 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:188773 10.3 10:c00d911b:::benchmark_data_smithi179_155357_object78277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:04.255173+0000", 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.842640461, 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23779645999999999, 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188773 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.014 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.255173+0000", 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.255173+0000", 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.255174+0000", 2024-03-24T11:36:09.015 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.136e-06 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.255178+0000", 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0539999999999996e-06 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.255189+0000", 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1063999999999999e-05 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.016 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.255201+0000", 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1797000000000001e-05 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.361070+0000", 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.105869634 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.361193+0000", 2024-03-24T11:36:09.017 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012298499999999999 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.361840+0000", 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00064659099999999996 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.368313+0000", 2024-03-24T11:36:09.018 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.006472574 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.374530+0000", 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0062174609999999996 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.492832+0000", 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.11830246 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.019 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.492904+0000", 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.1377000000000005e-05 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.492969+0000", 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.5327000000000007e-05 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:188775 10.3 10:c1b72ca3:::benchmark_data_smithi179_155357_object78279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:09.020 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:04.256068+0000", 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.8417448780000001, 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.23708838400000001, 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188775 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.256068+0000", 2024-03-24T11:36:09.021 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.256068+0000", 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.256069+0000", 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1900000000000001e-07 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.022 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.256073+0000", 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1350000000000002e-06 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.256087+0000", 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3186e-05 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.256094+0000", 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6559999999999992e-06 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.023 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.364874+0000", 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.108779917 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.365010+0000", 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000136389 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.365510+0000", 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00049935699999999997 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.024 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.374570+0000", 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0090604659999999997 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.374582+0000", 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1694e-05 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.493058+0000", 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.118475787 2024-03-24T11:36:09.025 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.493108+0000", 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0463000000000001e-05 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.493157+0000", 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8415000000000003e-05 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.026 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:188777 10.3 10:c5330514:::benchmark_data_smithi179_155357_object78281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:04.259470+0000", 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.8383431099999998, 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.234114818, 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 188777 2024-03-24T11:36:09.027 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.259470+0000", 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.259470+0000", 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.028 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.259471+0000", 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3629999999999999e-06 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.259477+0000", 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5199999999999997e-06 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.259488+0000", 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1158000000000001e-05 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.259496+0000", 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9869999999999999e-06 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.368339+0000", 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.10884297900000001 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.368462+0000", 2024-03-24T11:36:09.030 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012309800000000001 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 3,5", 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.368926+0000", 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00046394200000000001 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.374587+0000", 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056607150000000002 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.393896+0000", 2024-03-24T11:36:09.031 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.019308835999999999 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.493486+0000", 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.099589931000000007 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.493536+0000", 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0649000000000002e-05 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.032 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.493585+0000", 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8640000000000002e-05 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:189257 10.18 10:1853024e:::benchmark_data_smithi179_155357_object78761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:04.914674+0000", 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 1.183138939, 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15614108400000001, 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.033 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 189257 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.914674+0000", 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.034 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.914674+0000", 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.914676+0000", 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.593e-06 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.914682+0000", 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6999999999999996e-06 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.035 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.914697+0000", 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5569999999999998e-05 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.914706+0000", 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8030000000000003e-06 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.941981+0000", 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.027275020000000001 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.036 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.942106+0000", 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012488399999999999 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.942498+0000", 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00039176199999999998 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.945460+0000", 2024-03-24T11:36:09.037 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0029624130000000001 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:04.967063+0000", 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.021603513000000001 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.070671+0000", 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.103607846 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.038 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.070733+0000", 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1341999999999996e-05 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.070815+0000", 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2638000000000004e-05 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.039 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:189533 10.18 10:18f0c432:::benchmark_data_smithi179_155357_object79037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:05.323134+0000", 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.77467913399999999, 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.181849018, 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 189533 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.040 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323134+0000", 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323134+0000", 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323136+0000", 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5400000000000001e-06 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.041 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323141+0000", 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4700000000000001e-06 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323157+0000", 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5809999999999999e-05 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323165+0000", 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9619999999999993e-06 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.042 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323693+0000", 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00052849199999999998 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.323820+0000", 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000126976 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.324389+0000", 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056854599999999996 2024-03-24T11:36:09.043 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.343271+0000", 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018881737999999999 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.353249+0000", 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0099782470000000009 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.044 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.504864+0000", 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15161539800000001 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.504922+0000", 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7911999999999997e-05 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.504983+0000", 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0927000000000001e-05 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.045 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.15621.0:189564 10.8 e366/288 10:106bc265:::benchmark_data_smithi179_155357_object79068:head v 366'8487, mlcod=366'8485)", 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:05.368026+0000", 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.72978703499999997, 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15650228299999999, 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.046 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.368026+0000", 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.368026+0000", 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.368027+0000", 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4300000000000002e-07 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.047 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.368034+0000", 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.951e-06 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.368048+0000", 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4096e-05 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.368054+0000", 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.404e-06 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.048 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.506167+0000", 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13811268800000001 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.506185+0000", 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7543000000000001e-05 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.524488+0000", 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.018302952000000001 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.049 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.524528+0000", 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0905999999999998e-05 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:189803 10.18 10:19c56e95:::benchmark_data_smithi179_155357_object79307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:05.752693+0000", 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.34512014699999999, 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15840791000000001, 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.050 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 189803 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.752693+0000", 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.051 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.752693+0000", 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.752694+0000", 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.0800000000000003e-07 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.752699+0000", 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.639e-06 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.052 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.752710+0000", 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1148000000000001e-05 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.752717+0000", 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.498e-06 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.891930+0000", 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.139213219 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.892077+0000", 2024-03-24T11:36:09.053 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014694799999999999 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.892367+0000", 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028932099999999999 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.901062+0000", 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0086948949999999994 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.054 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.908587+0000", 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0075253439999999998 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.911006+0000", 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002419193 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.911055+0000", 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9005999999999998e-05 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.055 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.911101+0000", 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5791000000000003e-05 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:189820 10.18 10:19f24242:::benchmark_data_smithi179_155357_object79324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:05.782108+0000", 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.31570488699999999, 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15464929499999999, 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.056 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 189820 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.782108+0000", 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.782108+0000", 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.057 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.782110+0000", 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4440000000000001e-06 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.782114+0000", 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5140000000000003e-06 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.058 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.782123+0000", 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2229999999999996e-06 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.782130+0000", 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4119999999999998e-06 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.914113+0000", 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.13198284299999999 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.059 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.914246+0000", 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000133091 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.914814+0000", 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00056831899999999996 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.926088+0000", 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.011273447000000001 2024-03-24T11:36:09.060 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.926134+0000", 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6414e-05 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936652+0000", 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.01051819 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936699+0000", 2024-03-24T11:36:09.061 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6962999999999997e-05 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936758+0000", 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8434999999999999e-05 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:189822 10.18 10:1edd7f49:::benchmark_data_smithi179_155357_object79326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:05.784448+0000", 2024-03-24T11:36:09.062 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.313365065, 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.15240336099999999, 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 189822 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.784448+0000", 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.784448+0000", 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.784450+0000", 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3939999999999999e-06 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.784454+0000", 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.9230000000000001e-06 2024-03-24T11:36:09.064 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.784467+0000", 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2906e-05 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.784477+0000", 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2469999999999997e-06 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.922645+0000", 2024-03-24T11:36:09.065 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.138168765 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.922771+0000", 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000125957 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 1,3", 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.923519+0000", 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00074796900000000002 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.066 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936395+0000", 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.012876026 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936509+0000", 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00011347 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936765+0000", 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025642200000000002 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.067 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936805+0000", 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.9372999999999997e-05 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:05.936851+0000", 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6909000000000001e-05 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:09.068 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:36:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:09.070 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:09.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:36:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:36:09.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:09.210 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:09.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:09.261 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:36:09.267 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:36:09.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:09.351 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:09.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:09.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:09.491 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:09.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:09.506 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:09 smithi179 ceph-mon[29608]: pgmap v785: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-24T11:36:09.506 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:09 smithi179 ceph-mon[37907]: pgmap v785: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-24T11:36:09.506 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1550596291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:09.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:09.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:09.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:09.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:09.769 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:09.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:09.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:09.890 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1550596291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:09.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:09 smithi181 ceph-mon[33998]: pgmap v785: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-24T11:36:09.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1550596291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:09.910 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:09.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:10.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:10.049 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:10.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:10.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:10.185 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:10.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:10.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:10.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:10.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:10.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:10.463 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:10.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:10.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:36:10.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:10.616 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:10.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:10.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:10.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:10.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:10.813 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:10 smithi179 ceph-mon[29608]: 10.3 deep-scrub ok 2024-03-24T11:36:10.813 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:10 smithi179 ceph-mon[29608]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:36:10.814 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:10 smithi179 ceph-mon[37907]: 10.3 deep-scrub ok 2024-03-24T11:36:10.814 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:10 smithi179 ceph-mon[37907]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:36:10.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:10.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:10.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:10.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:10 smithi181 ceph-mon[33998]: 10.3 deep-scrub ok 2024-03-24T11:36:10.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:10 smithi181 ceph-mon[33998]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:36:10.986 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:11.026 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:11.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:11.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:11.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:11.187 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:11.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:11.190 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":785,"stamp":"2024-03-24T11:36:08.387581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2055149942,"num_objects":31549,"num_object_clones":0,"num_object_copies":94647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31549,"num_whiteouts":0,"num_read":1489,"num_read_kb":1241,"num_write":90741,"num_write_kb":3098798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2504,"num_bytes_recovered":162464294,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152035,"ondisk_log_size":152035,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":508,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11178040,"kb_used_data":6451164,"kb_used_omap":12,"kb_used_meta":4726579,"kb_avail":738783176,"statfs":{"total":767960285184,"available":756513972224,"internally_reserved":0,"allocated":6605991936,"data_stored":6602437742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4840017457},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,8,2,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-415525075,"num_objects":-6341,"num_object_clones":0,"num_object_copies":-19023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6341,"num_whiteouts":0,"num_read":1,"num_read_kb":-4,"num_write":-40080,"num_write_kb":-1766830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9648,"num_bytes_recovered":-607926905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002875"},"pg_stats":[{"pgid":"10.2","version":"367'8283","reported_seq":8522,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.056156+0000","last_change":"2024-03-24T11:35:57.243819+0000","last_active":"2024-03-24T11:36:08.056156+0000","last_peered":"2024-03-24T11:36:08.056156+0000","last_clean":"2024-03-24T11:36:08.056156+0000","last_became_active":"2024-03-24T11:33:46.312428+0000","last_became_peered":"2024-03-24T11:33:46.312428+0000","last_unstale":"2024-03-24T11:36:08.056156+0000","last_undegraded":"2024-03-24T11:36:08.056156+0000","last_fullsized":"2024-03-24T11:36:08.056156+0000","mapping_epoch":267,"log_start":"276'2138","ondisk_log_start":"276'2138","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'7990","last_scrub_stamp":"2024-03-24T11:35:57.243682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:57.243682+0000","objects_scrubbed":1207,"log_size":6145,"log_dups_size":0,"ondisk_log_size":6145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:13.427299+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59244544,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"213'20","reported_seq":694,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491818+0000","last_change":"2024-03-24T11:32:36.799202+0000","last_active":"2024-03-24T11:36:06.491818+0000","last_peered":"2024-03-24T11:36:06.491818+0000","last_clean":"2024-03-24T11:36:06.491818+0000","last_became_active":"2024-03-24T11:18:55.680499+0000","last_became_peered":"2024-03-24T11:18:55.680499+0000","last_unstale":"2024-03-24T11:36:06.491818+0000","last_undegraded":"2024-03-24T11:36:06.491818+0000","last_fullsized":"2024-03-24T11:36:06.491818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:32:36.799084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.799084+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":658,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.645826+0000","last_change":"2024-03-24T11:33:25.353540+0000","last_active":"2024-03-24T11:36:04.645826+0000","last_peered":"2024-03-24T11:36:04.645826+0000","last_clean":"2024-03-24T11:36:04.645826+0000","last_became_active":"2024-03-24T11:18:53.685178+0000","last_became_peered":"2024-03-24T11:18:53.685178+0000","last_unstale":"2024-03-24T11:36:04.645826+0000","last_undegraded":"2024-03-24T11:36:04.645826+0000","last_fullsized":"2024-03-24T11:36:04.645826+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:25.353054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:25.353054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"362'9","reported_seq":903,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.479400+0000","last_change":"2024-03-24T11:35:58.634213+0000","last_active":"2024-03-24T11:36:02.479400+0000","last_peered":"2024-03-24T11:36:02.479400+0000","last_clean":"2024-03-24T11:36:02.479400+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:36:02.479400+0000","last_undegraded":"2024-03-24T11:36:02.479400+0000","last_fullsized":"2024-03-24T11:36:02.479400+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'9","last_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:35:58.633956+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:07.079715+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":664,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.492065+0000","last_change":"2024-03-24T11:33:39.983218+0000","last_active":"2024-03-24T11:36:06.492065+0000","last_peered":"2024-03-24T11:36:06.492065+0000","last_clean":"2024-03-24T11:36:06.492065+0000","last_became_active":"2024-03-24T11:18:59.702745+0000","last_became_peered":"2024-03-24T11:18:59.702745+0000","last_unstale":"2024-03-24T11:36:06.492065+0000","last_undegraded":"2024-03-24T11:36:06.492065+0000","last_fullsized":"2024-03-24T11:36:06.492065+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_clean_scrub_stamp":"2024-03-24T11:33:39.980621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":734,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.620534+0000","last_change":"2024-03-24T11:31:10.289542+0000","last_active":"2024-03-24T11:36:04.620534+0000","last_peered":"2024-03-24T11:36:04.620534+0000","last_clean":"2024-03-24T11:36:04.620534+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:36:04.620534+0000","last_undegraded":"2024-03-24T11:36:04.620534+0000","last_fullsized":"2024-03-24T11:36:04.620534+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:10.289309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:31:10.289309+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"365'8700","reported_seq":8933,"reported_epoch":365,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T11:36:03.517919+0000","last_change":"2024-03-24T11:36:01.648085+0000","last_active":"2024-03-24T11:36:03.517919+0000","last_peered":"2024-03-24T11:36:03.517919+0000","last_clean":"2024-03-24T11:36:03.517919+0000","last_became_active":"2024-03-24T11:33:46.302266+0000","last_became_peered":"2024-03-24T11:33:46.302266+0000","last_unstale":"2024-03-24T11:36:03.517919+0000","last_undegraded":"2024-03-24T11:36:03.517919+0000","last_fullsized":"2024-03-24T11:36:03.517919+0000","mapping_epoch":267,"log_start":"274'1819","ondisk_log_start":"274'1819","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":274,"log_size":6881,"log_dups_size":0,"ondisk_log_size":6881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73990144,"num_objects":1129,"num_object_clones":0,"num_object_copies":3387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6003,"num_write_kb":228224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"213'39","reported_seq":753,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491900+0000","last_change":"2024-03-24T11:32:42.683845+0000","last_active":"2024-03-24T11:36:06.491900+0000","last_peered":"2024-03-24T11:36:06.491900+0000","last_clean":"2024-03-24T11:36:06.491900+0000","last_became_active":"2024-03-24T11:19:18.820366+0000","last_became_peered":"2024-03-24T11:19:18.820366+0000","last_unstale":"2024-03-24T11:36:06.491900+0000","last_undegraded":"2024-03-24T11:36:06.491900+0000","last_fullsized":"2024-03-24T11:36:06.491900+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:32:42.683738+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.683738+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":695,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478262+0000","last_change":"2024-03-24T11:32:23.906538+0000","last_active":"2024-03-24T11:36:02.478262+0000","last_peered":"2024-03-24T11:36:02.478262+0000","last_clean":"2024-03-24T11:36:02.478262+0000","last_became_active":"2024-03-24T11:18:53.682780+0000","last_became_peered":"2024-03-24T11:18:53.682780+0000","last_unstale":"2024-03-24T11:36:02.478262+0000","last_undegraded":"2024-03-24T11:36:02.478262+0000","last_fullsized":"2024-03-24T11:36:02.478262+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:32:23.906455+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.906455+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":676,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.510211+0000","last_change":"2024-03-24T11:32:25.619271+0000","last_active":"2024-03-24T11:36:06.510211+0000","last_peered":"2024-03-24T11:36:06.510211+0000","last_clean":"2024-03-24T11:36:06.510211+0000","last_became_active":"2024-03-24T11:18:57.699293+0000","last_became_peered":"2024-03-24T11:18:57.699293+0000","last_unstale":"2024-03-24T11:36:06.510211+0000","last_undegraded":"2024-03-24T11:36:06.510211+0000","last_fullsized":"2024-03-24T11:36:06.510211+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:25.619184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:32:25.619184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":676,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.483341+0000","last_change":"2024-03-24T11:33:26.128776+0000","last_active":"2024-03-24T11:36:02.483341+0000","last_peered":"2024-03-24T11:36:02.483341+0000","last_clean":"2024-03-24T11:36:02.483341+0000","last_became_active":"2024-03-24T11:18:59.709997+0000","last_became_peered":"2024-03-24T11:18:59.709997+0000","last_unstale":"2024-03-24T11:36:02.483341+0000","last_undegraded":"2024-03-24T11:36:02.483341+0000","last_fullsized":"2024-03-24T11:36:02.483341+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:26.128656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:26.128656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":837,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.486572+0000","last_change":"2024-03-24T11:36:02.213966+0000","last_active":"2024-03-24T11:36:02.486572+0000","last_peered":"2024-03-24T11:36:02.486572+0000","last_clean":"2024-03-24T11:36:02.486572+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:36:02.486572+0000","last_undegraded":"2024-03-24T11:36:02.486572+0000","last_fullsized":"2024-03-24T11:36:02.486572+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:36:02.213863+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:36:02.213863+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:12.421081+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"365'8030","reported_seq":8270,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.505361+0000","last_change":"2024-03-24T11:35:38.723007+0000","last_active":"2024-03-24T11:36:03.505361+0000","last_peered":"2024-03-24T11:36:03.505361+0000","last_clean":"2024-03-24T11:36:03.505361+0000","last_became_active":"2024-03-24T11:33:46.284906+0000","last_became_peered":"2024-03-24T11:33:46.284906+0000","last_unstale":"2024-03-24T11:36:03.505361+0000","last_undegraded":"2024-03-24T11:36:03.505361+0000","last_fullsized":"2024-03-24T11:36:03.505361+0000","mapping_epoch":267,"log_start":"273'1054","ondisk_log_start":"273'1054","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'7291","last_scrub_stamp":"2024-03-24T11:35:38.722871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:38.722871+0000","objects_scrubbed":1736,"log_size":6976,"log_dups_size":0,"ondisk_log_size":6976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:03.219267+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65339392,"num_objects":997,"num_object_clones":0,"num_object_copies":2991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"213'19","reported_seq":703,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491733+0000","last_change":"2024-03-24T11:32:37.784109+0000","last_active":"2024-03-24T11:36:06.491733+0000","last_peered":"2024-03-24T11:36:06.491733+0000","last_clean":"2024-03-24T11:36:06.491733+0000","last_became_active":"2024-03-24T11:18:55.684259+0000","last_became_peered":"2024-03-24T11:18:55.684259+0000","last_unstale":"2024-03-24T11:36:06.491733+0000","last_undegraded":"2024-03-24T11:36:06.491733+0000","last_fullsized":"2024-03-24T11:36:06.491733+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:37.784011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.784011+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":668,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.490954+0000","last_change":"2024-03-24T11:31:29.730215+0000","last_active":"2024-03-24T11:36:06.490954+0000","last_peered":"2024-03-24T11:36:06.490954+0000","last_clean":"2024-03-24T11:36:06.490954+0000","last_became_active":"2024-03-24T11:18:53.679183+0000","last_became_peered":"2024-03-24T11:18:53.679183+0000","last_unstale":"2024-03-24T11:36:06.490954+0000","last_undegraded":"2024-03-24T11:36:06.490954+0000","last_fullsized":"2024-03-24T11:36:06.490954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:29.730078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:29.730078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":664,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488266+0000","last_change":"2024-03-24T11:33:18.355226+0000","last_active":"2024-03-24T11:36:02.488266+0000","last_peered":"2024-03-24T11:36:02.488266+0000","last_clean":"2024-03-24T11:36:02.488266+0000","last_became_active":"2024-03-24T11:18:57.696497+0000","last_became_peered":"2024-03-24T11:18:57.696497+0000","last_unstale":"2024-03-24T11:36:02.488266+0000","last_undegraded":"2024-03-24T11:36:02.488266+0000","last_fullsized":"2024-03-24T11:36:02.488266+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:18.355107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:33:18.355107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.638847+0000","last_change":"2024-03-24T11:33:38.472649+0000","last_active":"2024-03-24T11:36:04.638847+0000","last_peered":"2024-03-24T11:36:04.638847+0000","last_clean":"2024-03-24T11:36:04.638847+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:36:04.638847+0000","last_undegraded":"2024-03-24T11:36:04.638847+0000","last_fullsized":"2024-03-24T11:36:04.638847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:38.472418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.472418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"367'8323","reported_seq":8560,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.188703+0000","last_change":"2024-03-24T11:36:00.549565+0000","last_active":"2024-03-24T11:36:08.188703+0000","last_peered":"2024-03-24T11:36:08.188703+0000","last_clean":"2024-03-24T11:36:08.188703+0000","last_became_active":"2024-03-24T11:33:46.277468+0000","last_became_peered":"2024-03-24T11:33:46.277468+0000","last_unstale":"2024-03-24T11:36:08.188703+0000","last_undegraded":"2024-03-24T11:36:08.188703+0000","last_fullsized":"2024-03-24T11:36:08.188703+0000","mapping_epoch":267,"log_start":"282'3745","ondisk_log_start":"282'3745","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'8122","last_scrub_stamp":"2024-03-24T11:36:00.549215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.549215+0000","objects_scrubbed":1081,"log_size":4578,"log_dups_size":0,"ondisk_log_size":4578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:03.267898+0000","scrub_duration":236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57475072,"num_objects":877,"num_object_clones":0,"num_object_copies":2631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"213'34","reported_seq":730,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.502263+0000","last_change":"2024-03-24T11:32:30.766638+0000","last_active":"2024-03-24T11:36:06.502263+0000","last_peered":"2024-03-24T11:36:06.502263+0000","last_clean":"2024-03-24T11:36:06.502263+0000","last_became_active":"2024-03-24T11:18:55.686127+0000","last_became_peered":"2024-03-24T11:18:55.686127+0000","last_unstale":"2024-03-24T11:36:06.502263+0000","last_undegraded":"2024-03-24T11:36:06.502263+0000","last_fullsized":"2024-03-24T11:36:06.502263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:32:30.766549+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.766549+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":658,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.623881+0000","last_change":"2024-03-24T11:31:22.167292+0000","last_active":"2024-03-24T11:36:04.623881+0000","last_peered":"2024-03-24T11:36:04.623881+0000","last_clean":"2024-03-24T11:36:04.623881+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:36:04.623881+0000","last_undegraded":"2024-03-24T11:36:04.623881+0000","last_fullsized":"2024-03-24T11:36:04.623881+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:22.167171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:31:22.167171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":875,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491965+0000","last_change":"2024-03-24T11:32:18.783487+0000","last_active":"2024-03-24T11:36:06.491965+0000","last_peered":"2024-03-24T11:36:06.491965+0000","last_clean":"2024-03-24T11:36:06.491965+0000","last_became_active":"2024-03-24T11:18:57.699066+0000","last_became_peered":"2024-03-24T11:18:57.699066+0000","last_unstale":"2024-03-24T11:36:06.491965+0000","last_undegraded":"2024-03-24T11:36:06.491965+0000","last_fullsized":"2024-03-24T11:36:06.491965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:32:18.783339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:18.783339+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":652,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487736+0000","last_change":"2024-03-24T11:31:28.690070+0000","last_active":"2024-03-24T11:36:02.487736+0000","last_peered":"2024-03-24T11:36:02.487736+0000","last_clean":"2024-03-24T11:36:02.487736+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:36:02.487736+0000","last_undegraded":"2024-03-24T11:36:02.487736+0000","last_fullsized":"2024-03-24T11:36:02.487736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:28.689946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:28.689946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":828,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487312+0000","last_change":"2024-03-24T11:33:21.341057+0000","last_active":"2024-03-24T11:36:02.487312+0000","last_peered":"2024-03-24T11:36:02.487312+0000","last_clean":"2024-03-24T11:36:02.487312+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:36:02.487312+0000","last_undegraded":"2024-03-24T11:36:02.487312+0000","last_fullsized":"2024-03-24T11:36:02.487312+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:21.340939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:21.340939+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"367'12095","reported_seq":12307,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.323673+0000","last_change":"2024-03-24T11:32:52.325471+0000","last_active":"2024-03-24T11:36:08.323673+0000","last_peered":"2024-03-24T11:36:08.323673+0000","last_clean":"2024-03-24T11:36:08.323673+0000","last_became_active":"2024-03-24T11:32:52.325161+0000","last_became_peered":"2024-03-24T11:32:52.325161+0000","last_unstale":"2024-03-24T11:36:08.323673+0000","last_undegraded":"2024-03-24T11:36:08.323673+0000","last_fullsized":"2024-03-24T11:36:08.323673+0000","mapping_epoch":267,"log_start":"305'7931","ondisk_log_start":"305'7931","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119865344,"num_objects":1829,"num_object_clones":0,"num_object_copies":5487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":445568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":669,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.490866+0000","last_change":"2024-03-24T11:32:39.715836+0000","last_active":"2024-03-24T11:36:06.490866+0000","last_peered":"2024-03-24T11:36:06.490866+0000","last_clean":"2024-03-24T11:36:06.490866+0000","last_became_active":"2024-03-24T11:18:57.698044+0000","last_became_peered":"2024-03-24T11:18:57.698044+0000","last_unstale":"2024-03-24T11:36:06.490866+0000","last_undegraded":"2024-03-24T11:36:06.490866+0000","last_fullsized":"2024-03-24T11:36:06.490866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.715759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"213'30","reported_seq":697,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.623970+0000","last_change":"2024-03-24T11:31:31.099444+0000","last_active":"2024-03-24T11:36:04.623970+0000","last_peered":"2024-03-24T11:36:04.623970+0000","last_clean":"2024-03-24T11:36:04.623970+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:36:04.623970+0000","last_undegraded":"2024-03-24T11:36:04.623970+0000","last_fullsized":"2024-03-24T11:36:04.623970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:31:31.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:31.099209+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":658,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.643767+0000","last_change":"2024-03-24T11:33:36.491136+0000","last_active":"2024-03-24T11:36:04.643767+0000","last_peered":"2024-03-24T11:36:04.643767+0000","last_clean":"2024-03-24T11:36:04.643767+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:36:04.643767+0000","last_undegraded":"2024-03-24T11:36:04.643767+0000","last_fullsized":"2024-03-24T11:36:04.643767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:33:36.491006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":657,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.490928+0000","last_change":"2024-03-24T11:33:30.982889+0000","last_active":"2024-03-24T11:36:06.490928+0000","last_peered":"2024-03-24T11:36:06.490928+0000","last_clean":"2024-03-24T11:36:06.490928+0000","last_became_active":"2024-03-24T11:18:59.705765+0000","last_became_peered":"2024-03-24T11:18:59.705765+0000","last_unstale":"2024-03-24T11:36:06.490928+0000","last_undegraded":"2024-03-24T11:36:06.490928+0000","last_fullsized":"2024-03-24T11:36:06.490928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:30.982770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:33:30.982770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":847,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491336+0000","last_change":"2024-03-24T11:35:48.392969+0000","last_active":"2024-03-24T11:36:06.491336+0000","last_peered":"2024-03-24T11:36:06.491336+0000","last_clean":"2024-03-24T11:36:06.491336+0000","last_became_active":"2024-03-24T11:35:48.392602+0000","last_became_peered":"2024-03-24T11:35:48.392602+0000","last_unstale":"2024-03-24T11:36:06.491336+0000","last_undegraded":"2024-03-24T11:36:06.491336+0000","last_fullsized":"2024-03-24T11:36:06.491336+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:31:25.692790+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:28:38.972499+0000","last_clean_scrub_stamp":"2024-03-24T11:31:25.692790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"365'11694","reported_seq":11904,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.519065+0000","last_change":"2024-03-24T11:32:52.673849+0000","last_active":"2024-03-24T11:36:03.519065+0000","last_peered":"2024-03-24T11:36:03.519065+0000","last_clean":"2024-03-24T11:36:03.519065+0000","last_became_active":"2024-03-24T11:32:52.673505+0000","last_became_peered":"2024-03-24T11:32:52.673505+0000","last_unstale":"2024-03-24T11:36:03.519065+0000","last_undegraded":"2024-03-24T11:36:03.519065+0000","last_fullsized":"2024-03-24T11:36:03.519065+0000","mapping_epoch":267,"log_start":"295'6502","ondisk_log_start":"295'6502","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132382720,"num_objects":2020,"num_object_clones":0,"num_object_copies":6060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11694,"num_write_kb":438848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":858,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.624061+0000","last_change":"2024-03-24T11:31:08.290147+0000","last_active":"2024-03-24T11:36:04.624061+0000","last_peered":"2024-03-24T11:36:04.624061+0000","last_clean":"2024-03-24T11:36:04.624061+0000","last_became_active":"2024-03-24T11:18:57.697236+0000","last_became_peered":"2024-03-24T11:18:57.697236+0000","last_unstale":"2024-03-24T11:36:04.624061+0000","last_undegraded":"2024-03-24T11:36:04.624061+0000","last_fullsized":"2024-03-24T11:36:04.624061+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:08.289960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:08.289960+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"213'28","reported_seq":707,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491100+0000","last_change":"2024-03-24T11:32:34.830458+0000","last_active":"2024-03-24T11:36:06.491100+0000","last_peered":"2024-03-24T11:36:06.491100+0000","last_clean":"2024-03-24T11:36:06.491100+0000","last_became_active":"2024-03-24T11:18:55.680772+0000","last_became_peered":"2024-03-24T11:18:55.680772+0000","last_unstale":"2024-03-24T11:36:06.491100+0000","last_undegraded":"2024-03-24T11:36:06.491100+0000","last_fullsized":"2024-03-24T11:36:06.491100+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:32:34.830322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:34.830322+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":673,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488260+0000","last_change":"2024-03-24T11:32:40.035107+0000","last_active":"2024-03-24T11:36:02.488260+0000","last_peered":"2024-03-24T11:36:02.488260+0000","last_clean":"2024-03-24T11:36:02.488260+0000","last_became_active":"2024-03-24T11:18:53.683115+0000","last_became_peered":"2024-03-24T11:18:53.683115+0000","last_unstale":"2024-03-24T11:36:02.488260+0000","last_undegraded":"2024-03-24T11:36:02.488260+0000","last_fullsized":"2024-03-24T11:36:02.488260+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:32:40.035022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.035022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":292,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478524+0000","last_change":"2024-03-24T11:35:58.443525+0000","last_active":"2024-03-24T11:36:02.478524+0000","last_peered":"2024-03-24T11:36:02.478524+0000","last_clean":"2024-03-24T11:36:02.478524+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:36:02.478524+0000","last_undegraded":"2024-03-24T11:36:02.478524+0000","last_fullsized":"2024-03-24T11:36:02.478524+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:58.443379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:58.443379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:25.302121+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":847,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.638676+0000","last_change":"2024-03-24T11:35:52.362580+0000","last_active":"2024-03-24T11:36:04.638676+0000","last_peered":"2024-03-24T11:36:04.638676+0000","last_clean":"2024-03-24T11:36:04.638676+0000","last_became_active":"2024-03-24T11:35:52.362283+0000","last_became_peered":"2024-03-24T11:35:52.362283+0000","last_unstale":"2024-03-24T11:36:04.638676+0000","last_undegraded":"2024-03-24T11:36:04.638676+0000","last_fullsized":"2024-03-24T11:36:04.638676+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:40.219272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:40.219272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"367'12025","reported_seq":12239,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.336781+0000","last_change":"2024-03-24T11:32:52.670936+0000","last_active":"2024-03-24T11:36:08.336781+0000","last_peered":"2024-03-24T11:36:08.336781+0000","last_clean":"2024-03-24T11:36:08.336781+0000","last_became_active":"2024-03-24T11:32:52.670591+0000","last_became_peered":"2024-03-24T11:32:52.670591+0000","last_unstale":"2024-03-24T11:36:08.336781+0000","last_undegraded":"2024-03-24T11:36:08.336781+0000","last_fullsized":"2024-03-24T11:36:08.336781+0000","mapping_epoch":267,"log_start":"305'7840","ondisk_log_start":"305'7840","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":5481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12025,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":880,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.490086+0000","last_change":"2024-03-24T11:32:42.033121+0000","last_active":"2024-03-24T11:36:02.490086+0000","last_peered":"2024-03-24T11:36:02.490086+0000","last_clean":"2024-03-24T11:36:02.490086+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:36:02.490086+0000","last_undegraded":"2024-03-24T11:36:02.490086+0000","last_fullsized":"2024-03-24T11:36:02.490086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.033019+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"213'20","reported_seq":719,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.503059+0000","last_change":"2024-03-24T11:36:03.939418+0000","last_active":"2024-03-24T11:36:06.503059+0000","last_peered":"2024-03-24T11:36:06.503059+0000","last_clean":"2024-03-24T11:36:06.503059+0000","last_became_active":"2024-03-24T11:18:55.686402+0000","last_became_peered":"2024-03-24T11:18:55.686402+0000","last_unstale":"2024-03-24T11:36:06.503059+0000","last_undegraded":"2024-03-24T11:36:06.503059+0000","last_fullsized":"2024-03-24T11:36:06.503059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:36:03.939287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:03.939287+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:32.618763+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":684,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488037+0000","last_change":"2024-03-24T11:33:42.448796+0000","last_active":"2024-03-24T11:36:02.488037+0000","last_peered":"2024-03-24T11:36:02.488037+0000","last_clean":"2024-03-24T11:36:02.488037+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:36:02.488037+0000","last_undegraded":"2024-03-24T11:36:02.488037+0000","last_fullsized":"2024-03-24T11:36:02.488037+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.448705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":654,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.485790+0000","last_change":"2024-03-24T11:32:22.994060+0000","last_active":"2024-03-24T11:36:02.485790+0000","last_peered":"2024-03-24T11:36:02.485790+0000","last_clean":"2024-03-24T11:36:02.485790+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:36:02.485790+0000","last_undegraded":"2024-03-24T11:36:02.485790+0000","last_fullsized":"2024-03-24T11:36:02.485790+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:32:22.993851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:22.993851+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":865,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.479064+0000","last_change":"2024-03-24T11:35:39.758485+0000","last_active":"2024-03-24T11:36:02.479064+0000","last_peered":"2024-03-24T11:36:02.479064+0000","last_clean":"2024-03-24T11:36:02.479064+0000","last_became_active":"2024-03-24T11:35:39.758022+0000","last_became_peered":"2024-03-24T11:35:39.758022+0000","last_unstale":"2024-03-24T11:36:02.479064+0000","last_undegraded":"2024-03-24T11:36:02.479064+0000","last_fullsized":"2024-03-24T11:36:02.479064+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:34.754561+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:33:34.754561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"365'11776","reported_seq":11987,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.496857+0000","last_change":"2024-03-24T11:32:52.327929+0000","last_active":"2024-03-24T11:36:03.496857+0000","last_peered":"2024-03-24T11:36:03.496857+0000","last_clean":"2024-03-24T11:36:03.496857+0000","last_became_active":"2024-03-24T11:32:52.327652+0000","last_became_peered":"2024-03-24T11:32:52.327652+0000","last_unstale":"2024-03-24T11:36:03.496857+0000","last_undegraded":"2024-03-24T11:36:03.496857+0000","last_fullsized":"2024-03-24T11:36:03.496857+0000","mapping_epoch":267,"log_start":"295'6575","ondisk_log_start":"295'6575","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132710424,"num_objects":2026,"num_object_clones":0,"num_object_copies":6078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11776,"num_write_kb":441601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":689,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478524+0000","last_change":"2024-03-24T11:33:33.804488+0000","last_active":"2024-03-24T11:36:02.478524+0000","last_peered":"2024-03-24T11:36:02.478524+0000","last_clean":"2024-03-24T11:36:02.478524+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:36:02.478524+0000","last_undegraded":"2024-03-24T11:36:02.478524+0000","last_fullsized":"2024-03-24T11:36:02.478524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:33.804384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.804384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"213'29","reported_seq":720,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.489788+0000","last_change":"2024-03-24T11:35:56.695685+0000","last_active":"2024-03-24T11:36:02.489788+0000","last_peered":"2024-03-24T11:36:02.489788+0000","last_clean":"2024-03-24T11:36:02.489788+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:36:02.489788+0000","last_undegraded":"2024-03-24T11:36:02.489788+0000","last_fullsized":"2024-03-24T11:36:02.489788+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:35:56.691809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:35:56.691809+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:02.020975+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":693,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.509848+0000","last_change":"2024-03-24T11:33:20.238699+0000","last_active":"2024-03-24T11:36:06.509848+0000","last_peered":"2024-03-24T11:36:06.509848+0000","last_clean":"2024-03-24T11:36:06.509848+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:36:06.509848+0000","last_undegraded":"2024-03-24T11:36:06.509848+0000","last_fullsized":"2024-03-24T11:36:06.509848+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:20.238585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.238585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":673,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.513789+0000","last_change":"2024-03-24T11:32:45.851666+0000","last_active":"2024-03-24T11:36:06.513789+0000","last_peered":"2024-03-24T11:36:06.513789+0000","last_clean":"2024-03-24T11:36:06.513789+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:36:06.513789+0000","last_undegraded":"2024-03-24T11:36:06.513789+0000","last_fullsized":"2024-03-24T11:36:06.513789+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.851566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.851566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":865,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478831+0000","last_change":"2024-03-24T11:35:43.803792+0000","last_active":"2024-03-24T11:36:02.478831+0000","last_peered":"2024-03-24T11:36:02.478831+0000","last_clean":"2024-03-24T11:36:02.478831+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:36:02.478831+0000","last_undegraded":"2024-03-24T11:36:02.478831+0000","last_fullsized":"2024-03-24T11:36:02.478831+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:32.751865+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:33:32.751865+0000","last_clean_scrub_stamp":"2024-03-24T11:33:32.751865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"365'8451","reported_seq":8688,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.445672+0000","last_change":"2024-03-24T11:35:59.169009+0000","last_active":"2024-03-24T11:36:03.445672+0000","last_peered":"2024-03-24T11:36:03.445672+0000","last_clean":"2024-03-24T11:36:03.445672+0000","last_became_active":"2024-03-24T11:33:46.285778+0000","last_became_peered":"2024-03-24T11:33:46.285778+0000","last_unstale":"2024-03-24T11:36:03.445672+0000","last_undegraded":"2024-03-24T11:36:03.445672+0000","last_fullsized":"2024-03-24T11:36:03.445672+0000","mapping_epoch":267,"log_start":"274'1875","ondisk_log_start":"274'1875","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'8308","last_scrub_stamp":"2024-03-24T11:35:59.165602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:59.165602+0000","objects_scrubbed":1228,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:09.900596+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70582272,"num_objects":1077,"num_object_clones":0,"num_object_copies":3231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":151,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":664,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488300+0000","last_change":"2024-03-24T11:33:35.552752+0000","last_active":"2024-03-24T11:36:02.488300+0000","last_peered":"2024-03-24T11:36:02.488300+0000","last_clean":"2024-03-24T11:36:02.488300+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:36:02.488300+0000","last_undegraded":"2024-03-24T11:36:02.488300+0000","last_fullsized":"2024-03-24T11:36:02.488300+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:35.552605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:33:35.552605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"213'24","reported_seq":214,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487703+0000","last_change":"2024-03-24T11:33:10.483025+0000","last_active":"2024-03-24T11:36:02.487703+0000","last_peered":"2024-03-24T11:36:02.487703+0000","last_clean":"2024-03-24T11:36:02.487703+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:36:02.487703+0000","last_undegraded":"2024-03-24T11:36:02.487703+0000","last_fullsized":"2024-03-24T11:36:02.487703+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:21.972644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:21.972644+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":663,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.606041+0000","last_change":"2024-03-24T11:32:39.822026+0000","last_active":"2024-03-24T11:36:06.606041+0000","last_peered":"2024-03-24T11:36:06.606041+0000","last_clean":"2024-03-24T11:36:06.606041+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:36:06.606041+0000","last_undegraded":"2024-03-24T11:36:06.606041+0000","last_fullsized":"2024-03-24T11:36:06.606041+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:32:39.821933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.821933+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":338,"num_read_kb":333,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":676,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.486560+0000","last_change":"2024-03-24T11:33:19.182254+0000","last_active":"2024-03-24T11:36:02.486560+0000","last_peered":"2024-03-24T11:36:02.486560+0000","last_clean":"2024-03-24T11:36:02.486560+0000","last_became_active":"2024-03-24T11:18:53.681219+0000","last_became_peered":"2024-03-24T11:18:53.681219+0000","last_unstale":"2024-03-24T11:36:02.486560+0000","last_undegraded":"2024-03-24T11:36:02.486560+0000","last_fullsized":"2024-03-24T11:36:02.486560+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.182060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.182060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":681,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.509570+0000","last_change":"2024-03-24T11:33:42.297388+0000","last_active":"2024-03-24T11:36:06.509570+0000","last_peered":"2024-03-24T11:36:06.509570+0000","last_clean":"2024-03-24T11:36:06.509570+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:36:06.509570+0000","last_undegraded":"2024-03-24T11:36:06.509570+0000","last_fullsized":"2024-03-24T11:36:06.509570+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.297267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.297267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":865,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.504690+0000","last_change":"2024-03-24T11:35:56.415820+0000","last_active":"2024-03-24T11:36:06.504690+0000","last_peered":"2024-03-24T11:36:06.504690+0000","last_clean":"2024-03-24T11:36:06.504690+0000","last_became_active":"2024-03-24T11:35:56.414826+0000","last_became_peered":"2024-03-24T11:35:56.414826+0000","last_unstale":"2024-03-24T11:36:06.504690+0000","last_undegraded":"2024-03-24T11:36:06.504690+0000","last_fullsized":"2024-03-24T11:36:06.504690+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:32:26.629894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:32:26.629894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"365'11609","reported_seq":11819,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.512222+0000","last_change":"2024-03-24T11:32:52.328051+0000","last_active":"2024-03-24T11:36:03.512222+0000","last_peered":"2024-03-24T11:36:03.512222+0000","last_clean":"2024-03-24T11:36:03.512222+0000","last_became_active":"2024-03-24T11:32:52.327768+0000","last_became_peered":"2024-03-24T11:32:52.327768+0000","last_unstale":"2024-03-24T11:36:03.512222+0000","last_undegraded":"2024-03-24T11:36:03.512222+0000","last_fullsized":"2024-03-24T11:36:03.512222+0000","mapping_epoch":267,"log_start":"295'6386","ondisk_log_start":"295'6386","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":5823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":433600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":284,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.482618+0000","last_change":"2024-03-24T11:32:41.207872+0000","last_active":"2024-03-24T11:36:02.482618+0000","last_peered":"2024-03-24T11:36:02.482618+0000","last_clean":"2024-03-24T11:36:02.482618+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:36:02.482618+0000","last_undegraded":"2024-03-24T11:36:02.482618+0000","last_fullsized":"2024-03-24T11:36:02.482618+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:41.207792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:32:41.207792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"213'36","reported_seq":947,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478795+0000","last_change":"2024-03-24T11:33:42.837753+0000","last_active":"2024-03-24T11:36:02.478795+0000","last_peered":"2024-03-24T11:36:02.478795+0000","last_clean":"2024-03-24T11:36:02.478795+0000","last_became_active":"2024-03-24T11:18:56.141183+0000","last_became_peered":"2024-03-24T11:18:56.141183+0000","last_unstale":"2024-03-24T11:36:02.478795+0000","last_undegraded":"2024-03-24T11:36:02.478795+0000","last_fullsized":"2024-03-24T11:36:02.478795+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'36","last_scrub_stamp":"2024-03-24T11:33:42.837610+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.837610+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":665,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.638619+0000","last_change":"2024-03-24T11:32:28.089047+0000","last_active":"2024-03-24T11:36:04.638619+0000","last_peered":"2024-03-24T11:36:04.638619+0000","last_clean":"2024-03-24T11:36:04.638619+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:36:04.638619+0000","last_undegraded":"2024-03-24T11:36:04.638619+0000","last_fullsized":"2024-03-24T11:36:04.638619+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.088969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.088969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":612,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.485892+0000","last_change":"2024-03-24T11:35:35.731208+0000","last_active":"2024-03-24T11:36:02.485892+0000","last_peered":"2024-03-24T11:36:02.485892+0000","last_clean":"2024-03-24T11:36:02.485892+0000","last_became_active":"2024-03-24T11:35:35.730443+0000","last_became_peered":"2024-03-24T11:35:35.730443+0000","last_unstale":"2024-03-24T11:36:02.485892+0000","last_undegraded":"2024-03-24T11:36:02.485892+0000","last_fullsized":"2024-03-24T11:36:02.485892+0000","mapping_epoch":237,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:33:18.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:33:18.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":676,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488050+0000","last_change":"2024-03-24T11:35:46.467497+0000","last_active":"2024-03-24T11:36:02.488050+0000","last_peered":"2024-03-24T11:36:02.488050+0000","last_clean":"2024-03-24T11:36:02.488050+0000","last_became_active":"2024-03-24T11:18:59.706846+0000","last_became_peered":"2024-03-24T11:18:59.706846+0000","last_unstale":"2024-03-24T11:36:02.488050+0000","last_undegraded":"2024-03-24T11:36:02.488050+0000","last_fullsized":"2024-03-24T11:36:02.488050+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:46.467403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:35:46.467403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:59.561947+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"365'8354","reported_seq":8595,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.524689+0000","last_change":"2024-03-24T11:35:38.595624+0000","last_active":"2024-03-24T11:36:03.524689+0000","last_peered":"2024-03-24T11:36:03.524689+0000","last_clean":"2024-03-24T11:36:03.524689+0000","last_became_active":"2024-03-24T11:33:46.279697+0000","last_became_peered":"2024-03-24T11:33:46.279697+0000","last_unstale":"2024-03-24T11:36:03.524689+0000","last_undegraded":"2024-03-24T11:36:03.524689+0000","last_fullsized":"2024-03-24T11:36:03.524689+0000","mapping_epoch":267,"log_start":"276'2343","ondisk_log_start":"276'2343","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'7583","last_scrub_stamp":"2024-03-24T11:35:38.595392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:38.595392+0000","objects_scrubbed":1766,"log_size":6011,"log_dups_size":0,"ondisk_log_size":6011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:53.032343+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65208320,"num_objects":995,"num_object_clones":0,"num_object_copies":2985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":771,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.479137+0000","last_change":"2024-03-24T11:33:24.909166+0000","last_active":"2024-03-24T11:36:02.479137+0000","last_peered":"2024-03-24T11:36:02.479137+0000","last_clean":"2024-03-24T11:36:02.479137+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:36:02.479137+0000","last_undegraded":"2024-03-24T11:36:02.479137+0000","last_fullsized":"2024-03-24T11:36:02.479137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:24.908806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.908806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"213'17","reported_seq":692,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.483162+0000","last_change":"2024-03-24T11:36:00.657669+0000","last_active":"2024-03-24T11:36:02.483162+0000","last_peered":"2024-03-24T11:36:02.483162+0000","last_clean":"2024-03-24T11:36:02.483162+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:36:02.483162+0000","last_undegraded":"2024-03-24T11:36:02.483162+0000","last_fullsized":"2024-03-24T11:36:02.483162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'17","last_scrub_stamp":"2024-03-24T11:36:00.657534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.657534+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:17.428989+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":698,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.479033+0000","last_change":"2024-03-24T11:33:29.862071+0000","last_active":"2024-03-24T11:36:02.479033+0000","last_peered":"2024-03-24T11:36:02.479033+0000","last_clean":"2024-03-24T11:36:02.479033+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:36:02.479033+0000","last_undegraded":"2024-03-24T11:36:02.479033+0000","last_fullsized":"2024-03-24T11:36:02.479033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:29.861850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:33:29.861850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":693,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.529165+0000","last_change":"2024-03-24T11:33:37.369520+0000","last_active":"2024-03-24T11:36:06.529165+0000","last_peered":"2024-03-24T11:36:06.529165+0000","last_clean":"2024-03-24T11:36:06.529165+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:36:06.529165+0000","last_undegraded":"2024-03-24T11:36:06.529165+0000","last_fullsized":"2024-03-24T11:36:06.529165+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:37.369254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:37.369254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":284,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478853+0000","last_change":"2024-03-24T11:32:42.994801+0000","last_active":"2024-03-24T11:36:02.478853+0000","last_peered":"2024-03-24T11:36:02.478853+0000","last_clean":"2024-03-24T11:36:02.478853+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:36:02.478853+0000","last_undegraded":"2024-03-24T11:36:02.478853+0000","last_fullsized":"2024-03-24T11:36:02.478853+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.994694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.994694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":620,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487127+0000","last_change":"2024-03-24T11:36:00.701664+0000","last_active":"2024-03-24T11:36:02.487127+0000","last_peered":"2024-03-24T11:36:02.487127+0000","last_clean":"2024-03-24T11:36:02.487127+0000","last_became_active":"2024-03-24T11:35:27.615367+0000","last_became_peered":"2024-03-24T11:35:27.615367+0000","last_unstale":"2024-03-24T11:36:02.487127+0000","last_undegraded":"2024-03-24T11:36:02.487127+0000","last_fullsized":"2024-03-24T11:36:02.487127+0000","mapping_epoch":237,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:00.701398+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.701398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:28.617384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"365'11865","reported_seq":12075,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.506865+0000","last_change":"2024-03-24T11:32:52.326526+0000","last_active":"2024-03-24T11:36:03.506865+0000","last_peered":"2024-03-24T11:36:03.506865+0000","last_clean":"2024-03-24T11:36:03.506865+0000","last_became_active":"2024-03-24T11:32:52.326215+0000","last_became_peered":"2024-03-24T11:32:52.326215+0000","last_unstale":"2024-03-24T11:36:03.506865+0000","last_undegraded":"2024-03-24T11:36:03.506865+0000","last_fullsized":"2024-03-24T11:36:03.506865+0000","mapping_epoch":267,"log_start":"295'6660","ondisk_log_start":"295'6660","created":267,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":0,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135987200,"num_objects":2075,"num_object_clones":0,"num_object_copies":6225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11865,"num_write_kb":446080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":665,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.491303+0000","last_change":"2024-03-24T11:33:33.162620+0000","last_active":"2024-03-24T11:36:02.491303+0000","last_peered":"2024-03-24T11:36:02.491303+0000","last_clean":"2024-03-24T11:36:02.491303+0000","last_became_active":"2024-03-24T11:18:57.694067+0000","last_became_peered":"2024-03-24T11:18:57.694067+0000","last_unstale":"2024-03-24T11:36:02.491303+0000","last_undegraded":"2024-03-24T11:36:02.491303+0000","last_fullsized":"2024-03-24T11:36:02.491303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:33.162392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.162392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"213'40","reported_seq":954,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.510031+0000","last_change":"2024-03-24T11:32:43.782374+0000","last_active":"2024-03-24T11:36:06.510031+0000","last_peered":"2024-03-24T11:36:06.510031+0000","last_clean":"2024-03-24T11:36:06.510031+0000","last_became_active":"2024-03-24T11:18:55.683504+0000","last_became_peered":"2024-03-24T11:18:55.683504+0000","last_unstale":"2024-03-24T11:36:06.510031+0000","last_undegraded":"2024-03-24T11:36:06.510031+0000","last_fullsized":"2024-03-24T11:36:06.510031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'40","last_scrub_stamp":"2024-03-24T11:32:43.782256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.782256+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":677,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491329+0000","last_change":"2024-03-24T11:35:57.272812+0000","last_active":"2024-03-24T11:36:06.491329+0000","last_peered":"2024-03-24T11:36:06.491329+0000","last_clean":"2024-03-24T11:36:06.491329+0000","last_became_active":"2024-03-24T11:18:53.680340+0000","last_became_peered":"2024-03-24T11:18:53.680340+0000","last_unstale":"2024-03-24T11:36:06.491329+0000","last_undegraded":"2024-03-24T11:36:06.491329+0000","last_fullsized":"2024-03-24T11:36:06.491329+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:57.272694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:57.272694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:02.752635+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":684,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487653+0000","last_change":"2024-03-24T11:32:28.707278+0000","last_active":"2024-03-24T11:36:02.487653+0000","last_peered":"2024-03-24T11:36:02.487653+0000","last_clean":"2024-03-24T11:36:02.487653+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:36:02.487653+0000","last_undegraded":"2024-03-24T11:36:02.487653+0000","last_fullsized":"2024-03-24T11:36:02.487653+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:32:28.707184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.707184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.639289+0000","last_change":"2024-03-24T11:33:35.480861+0000","last_active":"2024-03-24T11:36:04.639289+0000","last_peered":"2024-03-24T11:36:04.639289+0000","last_clean":"2024-03-24T11:36:04.639289+0000","last_became_active":"2024-03-24T11:18:59.705485+0000","last_became_peered":"2024-03-24T11:18:59.705485+0000","last_unstale":"2024-03-24T11:36:04.639289+0000","last_undegraded":"2024-03-24T11:36:04.639289+0000","last_fullsized":"2024-03-24T11:36:04.639289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:35.480687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:35.480687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":859,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.486551+0000","last_change":"2024-03-24T11:35:37.398831+0000","last_active":"2024-03-24T11:36:02.486551+0000","last_peered":"2024-03-24T11:36:02.486551+0000","last_clean":"2024-03-24T11:36:02.486551+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:36:02.486551+0000","last_undegraded":"2024-03-24T11:36:02.486551+0000","last_fullsized":"2024-03-24T11:36:02.486551+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:35:37.398700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:35:37.398700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:53.404682+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"365'8216","reported_seq":8450,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.320253+0000","last_change":"2024-03-24T11:36:02.990498+0000","last_active":"2024-03-24T11:36:03.320253+0000","last_peered":"2024-03-24T11:36:03.320253+0000","last_clean":"2024-03-24T11:36:03.320253+0000","last_became_active":"2024-03-24T11:33:46.315078+0000","last_became_peered":"2024-03-24T11:33:46.315078+0000","last_unstale":"2024-03-24T11:36:03.320253+0000","last_undegraded":"2024-03-24T11:36:03.320253+0000","last_fullsized":"2024-03-24T11:36:03.320253+0000","mapping_epoch":267,"log_start":"272'980","ondisk_log_start":"272'980","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'8201","last_scrub_stamp":"2024-03-24T11:36:02.990141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:36:02.990141+0000","objects_scrubbed":980,"log_size":7236,"log_dups_size":0,"ondisk_log_size":7236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:09.030626+0000","scrub_duration":322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63242240,"num_objects":965,"num_object_clones":0,"num_object_copies":2895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"213'15","reported_seq":693,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.490004+0000","last_change":"2024-03-24T11:32:37.060211+0000","last_active":"2024-03-24T11:36:02.490004+0000","last_peered":"2024-03-24T11:36:02.490004+0000","last_clean":"2024-03-24T11:36:02.490004+0000","last_became_active":"2024-03-24T11:18:55.683825+0000","last_became_peered":"2024-03-24T11:18:55.683825+0000","last_unstale":"2024-03-24T11:36:02.490004+0000","last_undegraded":"2024-03-24T11:36:02.490004+0000","last_fullsized":"2024-03-24T11:36:02.490004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:32:37.060067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:32:37.060067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":686,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488450+0000","last_change":"2024-03-24T11:35:52.267351+0000","last_active":"2024-03-24T11:36:02.488450+0000","last_peered":"2024-03-24T11:36:02.488450+0000","last_clean":"2024-03-24T11:36:02.488450+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:36:02.488450+0000","last_undegraded":"2024-03-24T11:36:02.488450+0000","last_fullsized":"2024-03-24T11:36:02.488450+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:35:52.267270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:52.267270+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:20.415155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.479470+0000","last_change":"2024-03-24T11:33:37.859003+0000","last_active":"2024-03-24T11:36:02.479470+0000","last_peered":"2024-03-24T11:36:02.479470+0000","last_clean":"2024-03-24T11:36:02.479470+0000","last_became_active":"2024-03-24T11:18:57.699554+0000","last_became_peered":"2024-03-24T11:18:57.699554+0000","last_unstale":"2024-03-24T11:36:02.479470+0000","last_undegraded":"2024-03-24T11:36:02.479470+0000","last_fullsized":"2024-03-24T11:36:02.479470+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:37.858877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:37.858877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.486781+0000","last_change":"2024-03-24T11:33:37.314906+0000","last_active":"2024-03-24T11:36:02.486781+0000","last_peered":"2024-03-24T11:36:02.486781+0000","last_clean":"2024-03-24T11:36:02.486781+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:36:02.486781+0000","last_undegraded":"2024-03-24T11:36:02.486781+0000","last_fullsized":"2024-03-24T11:36:02.486781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:37.314690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:37.314690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"365'8247","reported_seq":8485,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.390414+0000","last_change":"2024-03-24T11:36:00.032835+0000","last_active":"2024-03-24T11:36:03.390414+0000","last_peered":"2024-03-24T11:36:03.390414+0000","last_clean":"2024-03-24T11:36:03.390414+0000","last_became_active":"2024-03-24T11:33:46.308480+0000","last_became_peered":"2024-03-24T11:33:46.308480+0000","last_unstale":"2024-03-24T11:36:03.390414+0000","last_undegraded":"2024-03-24T11:36:03.390414+0000","last_fullsized":"2024-03-24T11:36:03.390414+0000","mapping_epoch":267,"log_start":"274'1521","ondisk_log_start":"274'1521","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'8153","last_scrub_stamp":"2024-03-24T11:36:00.032574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.032574+0000","objects_scrubbed":1097,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:15.350639+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65339392,"num_objects":997,"num_object_clones":0,"num_object_copies":2991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"213'44","reported_seq":748,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478589+0000","last_change":"2024-03-24T11:33:24.826729+0000","last_active":"2024-03-24T11:36:02.478589+0000","last_peered":"2024-03-24T11:36:02.478589+0000","last_clean":"2024-03-24T11:36:02.478589+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:36:02.478589+0000","last_undegraded":"2024-03-24T11:36:02.478589+0000","last_fullsized":"2024-03-24T11:36:02.478589+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:33:24.826519+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.826519+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":276,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488198+0000","last_change":"2024-03-24T11:32:38.827248+0000","last_active":"2024-03-24T11:36:02.488198+0000","last_peered":"2024-03-24T11:36:02.488198+0000","last_clean":"2024-03-24T11:36:02.488198+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:36:02.488198+0000","last_undegraded":"2024-03-24T11:36:02.488198+0000","last_fullsized":"2024-03-24T11:36:02.488198+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.827167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.827167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.501671+0000","last_change":"2024-03-24T11:33:27.894256+0000","last_active":"2024-03-24T11:36:02.501671+0000","last_peered":"2024-03-24T11:36:02.501671+0000","last_clean":"2024-03-24T11:36:02.501671+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:36:02.501671+0000","last_undegraded":"2024-03-24T11:36:02.501671+0000","last_fullsized":"2024-03-24T11:36:02.501671+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:33:27.894028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":284,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.483705+0000","last_change":"2024-03-24T11:33:22.957967+0000","last_active":"2024-03-24T11:36:02.483705+0000","last_peered":"2024-03-24T11:36:02.483705+0000","last_clean":"2024-03-24T11:36:02.483705+0000","last_became_active":"2024-03-24T11:30:35.355210+0000","last_became_peered":"2024-03-24T11:30:35.355210+0000","last_unstale":"2024-03-24T11:36:02.483705+0000","last_undegraded":"2024-03-24T11:36:02.483705+0000","last_fullsized":"2024-03-24T11:36:02.483705+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:22.957812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.957812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.7","version":"365'8202","reported_seq":8445,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.525486+0000","last_change":"2024-03-24T11:35:40.033049+0000","last_active":"2024-03-24T11:36:03.525486+0000","last_peered":"2024-03-24T11:36:03.525486+0000","last_clean":"2024-03-24T11:36:03.525486+0000","last_became_active":"2024-03-24T11:33:46.296101+0000","last_became_peered":"2024-03-24T11:33:46.296101+0000","last_unstale":"2024-03-24T11:36:03.525486+0000","last_undegraded":"2024-03-24T11:36:03.525486+0000","last_fullsized":"2024-03-24T11:36:03.525486+0000","mapping_epoch":267,"log_start":"274'1381","ondisk_log_start":"274'1381","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'7494","last_scrub_stamp":"2024-03-24T11:35:40.032696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:40.032696+0000","objects_scrubbed":1739,"log_size":6821,"log_dups_size":0,"ondisk_log_size":6821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:51.031169+0000","scrub_duration":495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66912256,"num_objects":1021,"num_object_clones":0,"num_object_copies":3063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.483463+0000","last_change":"2024-03-24T11:32:44.207912+0000","last_active":"2024-03-24T11:36:02.483463+0000","last_peered":"2024-03-24T11:36:02.483463+0000","last_clean":"2024-03-24T11:36:02.483463+0000","last_became_active":"2024-03-24T11:18:57.703789+0000","last_became_peered":"2024-03-24T11:18:57.703789+0000","last_unstale":"2024-03-24T11:36:02.483463+0000","last_undegraded":"2024-03-24T11:36:02.483463+0000","last_fullsized":"2024-03-24T11:36:02.483463+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:44.207817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:44.207817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"213'18","reported_seq":704,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.492920+0000","last_change":"2024-03-24T11:35:58.274112+0000","last_active":"2024-03-24T11:36:06.492920+0000","last_peered":"2024-03-24T11:36:06.492920+0000","last_clean":"2024-03-24T11:36:06.492920+0000","last_became_active":"2024-03-24T11:18:55.682793+0000","last_became_peered":"2024-03-24T11:18:55.682793+0000","last_unstale":"2024-03-24T11:36:06.492920+0000","last_undegraded":"2024-03-24T11:36:06.492920+0000","last_fullsized":"2024-03-24T11:36:06.492920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:35:58.273980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:35:58.273980+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:12.721302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":280,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491615+0000","last_change":"2024-03-24T11:33:20.771559+0000","last_active":"2024-03-24T11:36:06.491615+0000","last_peered":"2024-03-24T11:36:06.491615+0000","last_clean":"2024-03-24T11:36:06.491615+0000","last_became_active":"2024-03-24T11:30:35.356055+0000","last_became_peered":"2024-03-24T11:30:35.356055+0000","last_unstale":"2024-03-24T11:36:06.491615+0000","last_undegraded":"2024-03-24T11:36:06.491615+0000","last_fullsized":"2024-03-24T11:36:06.491615+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:20.771417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:33:20.771417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.646124+0000","last_change":"2024-03-24T11:33:42.747143+0000","last_active":"2024-03-24T11:36:04.646124+0000","last_peered":"2024-03-24T11:36:04.646124+0000","last_clean":"2024-03-24T11:36:04.646124+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:36:04.646124+0000","last_undegraded":"2024-03-24T11:36:04.646124+0000","last_fullsized":"2024-03-24T11:36:04.646124+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.746752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.746752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"365'8343","reported_seq":8588,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.598092+0000","last_change":"2024-03-24T11:35:39.862906+0000","last_active":"2024-03-24T11:36:03.598092+0000","last_peered":"2024-03-24T11:36:03.598092+0000","last_clean":"2024-03-24T11:36:03.598092+0000","last_became_active":"2024-03-24T11:33:46.315487+0000","last_became_peered":"2024-03-24T11:33:46.315487+0000","last_unstale":"2024-03-24T11:36:03.598092+0000","last_undegraded":"2024-03-24T11:36:03.598092+0000","last_fullsized":"2024-03-24T11:36:03.598092+0000","mapping_epoch":267,"log_start":"276'2280","ondisk_log_start":"276'2280","created":267,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"343'7613","last_scrub_stamp":"2024-03-24T11:35:39.862619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:39.862619+0000","objects_scrubbed":1785,"log_size":6063,"log_dups_size":0,"ondisk_log_size":6063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:09.206174+0000","scrub_duration":518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68354048,"num_objects":1043,"num_object_clones":0,"num_object_copies":3129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"357'10","reported_seq":384,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478126+0000","last_change":"2024-03-24T11:35:53.387025+0000","last_active":"2024-03-24T11:36:02.478126+0000","last_peered":"2024-03-24T11:36:02.478126+0000","last_clean":"2024-03-24T11:36:02.478126+0000","last_became_active":"2024-03-24T11:30:35.359635+0000","last_became_peered":"2024-03-24T11:30:35.359635+0000","last_unstale":"2024-03-24T11:36:02.478126+0000","last_undegraded":"2024-03-24T11:36:02.478126+0000","last_fullsized":"2024-03-24T11:36:02.478126+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"357'10","last_scrub_stamp":"2024-03-24T11:35:53.386928+0000","last_deep_scrub":"357'10","last_deep_scrub_stamp":"2024-03-24T11:35:53.386928+0000","last_clean_scrub_stamp":"2024-03-24T11:35:53.386928+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:06.281135+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"213'34","reported_seq":726,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.486329+0000","last_change":"2024-03-24T11:35:51.276943+0000","last_active":"2024-03-24T11:36:02.486329+0000","last_peered":"2024-03-24T11:36:02.486329+0000","last_clean":"2024-03-24T11:36:02.486329+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:36:02.486329+0000","last_undegraded":"2024-03-24T11:36:02.486329+0000","last_fullsized":"2024-03-24T11:36:02.486329+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:35:51.276843+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:35:51.276843+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:55.861618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":482,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491827+0000","last_change":"2024-03-24T11:33:22.820364+0000","last_active":"2024-03-24T11:36:06.491827+0000","last_peered":"2024-03-24T11:36:06.491827+0000","last_clean":"2024-03-24T11:36:06.491827+0000","last_became_active":"2024-03-24T11:30:35.356100+0000","last_became_peered":"2024-03-24T11:30:35.356100+0000","last_unstale":"2024-03-24T11:36:06.491827+0000","last_undegraded":"2024-03-24T11:36:06.491827+0000","last_fullsized":"2024-03-24T11:36:06.491827+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:33:22.820253+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.820253+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":668,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487966+0000","last_change":"2024-03-24T11:35:48.394162+0000","last_active":"2024-03-24T11:36:02.487966+0000","last_peered":"2024-03-24T11:36:02.487966+0000","last_clean":"2024-03-24T11:36:02.487966+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:36:02.487966+0000","last_undegraded":"2024-03-24T11:36:02.487966+0000","last_fullsized":"2024-03-24T11:36:02.487966+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:48.394040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:48.394040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:16.608556+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.19","version":"367'8471","reported_seq":18363,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.043997+0000","last_change":"2024-03-24T11:35:59.240784+0000","last_active":"2024-03-24T11:36:08.043997+0000","last_peered":"2024-03-24T11:36:08.043997+0000","last_clean":"2024-03-24T11:36:08.043997+0000","last_became_active":"2024-03-24T11:35:46.282249+0000","last_became_peered":"2024-03-24T11:35:46.282249+0000","last_unstale":"2024-03-24T11:36:08.043997+0000","last_undegraded":"2024-03-24T11:36:08.043997+0000","last_fullsized":"2024-03-24T11:36:08.043997+0000","mapping_epoch":349,"log_start":"276'2484","ondisk_log_start":"276'2484","created":288,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'8209","last_scrub_stamp":"2024-03-24T11:35:59.240617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:59.240617+0000","objects_scrubbed":1209,"log_size":5987,"log_dups_size":0,"ondisk_log_size":5987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:07.102264+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61472768,"num_objects":938,"num_object_clones":0,"num_object_copies":2814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":271,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"213'15","reported_seq":675,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.640348+0000","last_change":"2024-03-24T11:35:53.799172+0000","last_active":"2024-03-24T11:36:04.640348+0000","last_peered":"2024-03-24T11:36:04.640348+0000","last_clean":"2024-03-24T11:36:04.640348+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:36:04.640348+0000","last_undegraded":"2024-03-24T11:36:04.640348+0000","last_fullsized":"2024-03-24T11:36:04.640348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:35:53.797708+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:35:53.797708+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:58.814256+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":696,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.477850+0000","last_change":"2024-03-24T11:35:42.517251+0000","last_active":"2024-03-24T11:36:02.477850+0000","last_peered":"2024-03-24T11:36:02.477850+0000","last_clean":"2024-03-24T11:36:02.477850+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:36:02.477850+0000","last_undegraded":"2024-03-24T11:36:02.477850+0000","last_fullsized":"2024-03-24T11:36:02.477850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:42.517142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:42.517142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:11.043366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":664,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487769+0000","last_change":"2024-03-24T11:32:39.835187+0000","last_active":"2024-03-24T11:36:02.487769+0000","last_peered":"2024-03-24T11:36:02.487769+0000","last_clean":"2024-03-24T11:36:02.487769+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:36:02.487769+0000","last_undegraded":"2024-03-24T11:36:02.487769+0000","last_fullsized":"2024-03-24T11:36:02.487769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:39.835097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:39.835097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":292,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478908+0000","last_change":"2024-03-24T11:33:36.829894+0000","last_active":"2024-03-24T11:36:02.478908+0000","last_peered":"2024-03-24T11:36:02.478908+0000","last_clean":"2024-03-24T11:36:02.478908+0000","last_became_active":"2024-03-24T11:30:35.353212+0000","last_became_peered":"2024-03-24T11:30:35.353212+0000","last_unstale":"2024-03-24T11:36:02.478908+0000","last_undegraded":"2024-03-24T11:36:02.478908+0000","last_fullsized":"2024-03-24T11:36:02.478908+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:36.829776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:33:36.829776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.18","version":"365'8897","reported_seq":16571,"reported_epoch":365,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T11:36:03.505437+0000","last_change":"2024-03-24T11:35:59.605786+0000","last_active":"2024-03-24T11:36:03.505437+0000","last_peered":"2024-03-24T11:36:03.505437+0000","last_clean":"2024-03-24T11:36:03.505437+0000","last_became_active":"2024-03-24T11:34:57.313113+0000","last_became_peered":"2024-03-24T11:34:57.313113+0000","last_unstale":"2024-03-24T11:36:03.505437+0000","last_undegraded":"2024-03-24T11:36:03.505437+0000","last_fullsized":"2024-03-24T11:36:03.505437+0000","mapping_epoch":308,"log_start":"276'2236","ondisk_log_start":"276'2236","created":288,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:32:51.307286+0000","objects_scrubbed":542,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65208320,"num_objects":995,"num_object_clones":0,"num_object_copies":2985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5929,"num_write_kb":221568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2474,"num_bytes_recovered":162070528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"213'19","reported_seq":697,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488466+0000","last_change":"2024-03-24T11:32:32.023601+0000","last_active":"2024-03-24T11:36:02.488466+0000","last_peered":"2024-03-24T11:36:02.488466+0000","last_clean":"2024-03-24T11:36:02.488466+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:36:02.488466+0000","last_undegraded":"2024-03-24T11:36:02.488466+0000","last_fullsized":"2024-03-24T11:36:02.488466+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:32:32.023502+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.023502+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":280,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491215+0000","last_change":"2024-03-24T11:33:33.927309+0000","last_active":"2024-03-24T11:36:06.491215+0000","last_peered":"2024-03-24T11:36:06.491215+0000","last_clean":"2024-03-24T11:36:06.491215+0000","last_became_active":"2024-03-24T11:30:35.358928+0000","last_became_peered":"2024-03-24T11:30:35.358928+0000","last_unstale":"2024-03-24T11:36:06.491215+0000","last_undegraded":"2024-03-24T11:36:06.491215+0000","last_fullsized":"2024-03-24T11:36:06.491215+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:33.927191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.927191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":650,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.640265+0000","last_change":"2024-03-24T11:33:32.456675+0000","last_active":"2024-03-24T11:36:04.640265+0000","last_peered":"2024-03-24T11:36:04.640265+0000","last_clean":"2024-03-24T11:36:04.640265+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:36:04.640265+0000","last_undegraded":"2024-03-24T11:36:04.640265+0000","last_fullsized":"2024-03-24T11:36:04.640265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:32.456539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:33:32.456539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":676,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.479228+0000","last_change":"2024-03-24T11:35:57.646522+0000","last_active":"2024-03-24T11:36:02.479228+0000","last_peered":"2024-03-24T11:36:02.479228+0000","last_clean":"2024-03-24T11:36:02.479228+0000","last_became_active":"2024-03-24T11:18:59.702711+0000","last_became_peered":"2024-03-24T11:18:59.702711+0000","last_unstale":"2024-03-24T11:36:02.479228+0000","last_undegraded":"2024-03-24T11:36:02.479228+0000","last_fullsized":"2024-03-24T11:36:02.479228+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:57.646417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:57.646417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:01.236188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"213'15","reported_seq":693,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488291+0000","last_change":"2024-03-24T11:31:27.541622+0000","last_active":"2024-03-24T11:36:02.488291+0000","last_peered":"2024-03-24T11:36:02.488291+0000","last_clean":"2024-03-24T11:36:02.488291+0000","last_became_active":"2024-03-24T11:18:56.142972+0000","last_became_peered":"2024-03-24T11:18:56.142972+0000","last_unstale":"2024-03-24T11:36:02.488291+0000","last_undegraded":"2024-03-24T11:36:02.488291+0000","last_fullsized":"2024-03-24T11:36:02.488291+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:31:27.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:27.541510+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":672,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487051+0000","last_change":"2024-03-24T11:32:45.985820+0000","last_active":"2024-03-24T11:36:02.487051+0000","last_peered":"2024-03-24T11:36:02.487051+0000","last_clean":"2024-03-24T11:36:02.487051+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:36:02.487051+0000","last_undegraded":"2024-03-24T11:36:02.487051+0000","last_fullsized":"2024-03-24T11:36:02.487051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.985741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:46.640304+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.985741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":865,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.639664+0000","last_change":"2024-03-24T11:31:13.201069+0000","last_active":"2024-03-24T11:36:04.639664+0000","last_peered":"2024-03-24T11:36:04.639664+0000","last_clean":"2024-03-24T11:36:04.639664+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:36:04.639664+0000","last_undegraded":"2024-03-24T11:36:04.639664+0000","last_fullsized":"2024-03-24T11:36:04.639664+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:13.200503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:13.200503+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":656,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491995+0000","last_change":"2024-03-24T11:32:27.764304+0000","last_active":"2024-03-24T11:36:06.491995+0000","last_peered":"2024-03-24T11:36:06.491995+0000","last_clean":"2024-03-24T11:36:06.491995+0000","last_became_active":"2024-03-24T11:18:59.705445+0000","last_became_peered":"2024-03-24T11:18:59.705445+0000","last_unstale":"2024-03-24T11:36:06.491995+0000","last_undegraded":"2024-03-24T11:36:06.491995+0000","last_fullsized":"2024-03-24T11:36:06.491995+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:27.764203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:32:27.764203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"213'20","reported_seq":684,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.639513+0000","last_change":"2024-03-24T11:33:19.230202+0000","last_active":"2024-03-24T11:36:04.639513+0000","last_peered":"2024-03-24T11:36:04.639513+0000","last_clean":"2024-03-24T11:36:04.639513+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:36:04.639513+0000","last_undegraded":"2024-03-24T11:36:04.639513+0000","last_fullsized":"2024-03-24T11:36:04.639513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:19.229920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.229920+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":280,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491566+0000","last_change":"2024-03-24T11:31:30.728130+0000","last_active":"2024-03-24T11:36:06.491566+0000","last_peered":"2024-03-24T11:36:06.491566+0000","last_clean":"2024-03-24T11:36:06.491566+0000","last_became_active":"2024-03-24T11:30:35.353946+0000","last_became_peered":"2024-03-24T11:30:35.353946+0000","last_unstale":"2024-03-24T11:36:06.491566+0000","last_undegraded":"2024-03-24T11:36:06.491566+0000","last_fullsized":"2024-03-24T11:36:06.491566+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:30.728004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:30.728004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":878,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487936+0000","last_change":"2024-03-24T11:31:36.682644+0000","last_active":"2024-03-24T11:36:02.487936+0000","last_peered":"2024-03-24T11:36:02.487936+0000","last_clean":"2024-03-24T11:36:02.487936+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:36:02.487936+0000","last_undegraded":"2024-03-24T11:36:02.487936+0000","last_fullsized":"2024-03-24T11:36:02.487936+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:31:36.682542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:31:36.682542+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":652,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488903+0000","last_change":"2024-03-24T11:32:31.071719+0000","last_active":"2024-03-24T11:36:02.488903+0000","last_peered":"2024-03-24T11:36:02.488903+0000","last_clean":"2024-03-24T11:36:02.488903+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:36:02.488903+0000","last_undegraded":"2024-03-24T11:36:02.488903+0000","last_fullsized":"2024-03-24T11:36:02.488903+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:31.071621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:31.071621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"213'24","reported_seq":708,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491652+0000","last_change":"2024-03-24T11:32:43.646863+0000","last_active":"2024-03-24T11:36:06.491652+0000","last_peered":"2024-03-24T11:36:06.491652+0000","last_clean":"2024-03-24T11:36:06.491652+0000","last_became_active":"2024-03-24T11:18:55.686072+0000","last_became_peered":"2024-03-24T11:18:55.686072+0000","last_unstale":"2024-03-24T11:36:06.491652+0000","last_undegraded":"2024-03-24T11:36:06.491652+0000","last_fullsized":"2024-03-24T11:36:06.491652+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:43.646773+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.646773+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":668,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491666+0000","last_change":"2024-03-24T11:31:37.636078+0000","last_active":"2024-03-24T11:36:06.491666+0000","last_peered":"2024-03-24T11:36:06.491666+0000","last_clean":"2024-03-24T11:36:06.491666+0000","last_became_active":"2024-03-24T11:18:53.680480+0000","last_became_peered":"2024-03-24T11:18:53.680480+0000","last_unstale":"2024-03-24T11:36:06.491666+0000","last_undegraded":"2024-03-24T11:36:06.491666+0000","last_fullsized":"2024-03-24T11:36:06.491666+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:37.635705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:31:37.635705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":684,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.502982+0000","last_change":"2024-03-24T11:32:36.752484+0000","last_active":"2024-03-24T11:36:06.502982+0000","last_peered":"2024-03-24T11:36:06.502982+0000","last_clean":"2024-03-24T11:36:06.502982+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:36:06.502982+0000","last_undegraded":"2024-03-24T11:36:06.502982+0000","last_fullsized":"2024-03-24T11:36:06.502982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:36.752406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:32:36.752406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.638305+0000","last_change":"2024-03-24T11:32:32.077273+0000","last_active":"2024-03-24T11:36:04.638305+0000","last_peered":"2024-03-24T11:36:04.638305+0000","last_clean":"2024-03-24T11:36:04.638305+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:36:04.638305+0000","last_undegraded":"2024-03-24T11:36:04.638305+0000","last_fullsized":"2024-03-24T11:36:04.638305+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:32.077150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:32.077150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"213'15","reported_seq":705,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.503142+0000","last_change":"2024-03-24T11:33:28.304466+0000","last_active":"2024-03-24T11:36:06.503142+0000","last_peered":"2024-03-24T11:36:06.503142+0000","last_clean":"2024-03-24T11:36:06.503142+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:36:06.503142+0000","last_undegraded":"2024-03-24T11:36:06.503142+0000","last_fullsized":"2024-03-24T11:36:06.503142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:33:28.304078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:28.304078+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":278,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.639015+0000","last_change":"2024-03-24T11:32:45.875224+0000","last_active":"2024-03-24T11:36:04.639015+0000","last_peered":"2024-03-24T11:36:04.639015+0000","last_clean":"2024-03-24T11:36:04.639015+0000","last_became_active":"2024-03-24T11:30:35.353230+0000","last_became_peered":"2024-03-24T11:30:35.353230+0000","last_unstale":"2024-03-24T11:36:04.639015+0000","last_undegraded":"2024-03-24T11:36:04.639015+0000","last_fullsized":"2024-03-24T11:36:04.639015+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:45.875145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:45.875145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":650,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.645944+0000","last_change":"2024-03-24T11:33:26.388166+0000","last_active":"2024-03-24T11:36:04.645944+0000","last_peered":"2024-03-24T11:36:04.645944+0000","last_clean":"2024-03-24T11:36:04.645944+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:36:04.645944+0000","last_undegraded":"2024-03-24T11:36:04.645944+0000","last_fullsized":"2024-03-24T11:36:04.645944+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:26.387703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:26.387703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.485679+0000","last_change":"2024-03-24T11:35:57.745325+0000","last_active":"2024-03-24T11:36:02.485679+0000","last_peered":"2024-03-24T11:36:02.485679+0000","last_clean":"2024-03-24T11:36:02.485679+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:36:02.485679+0000","last_undegraded":"2024-03-24T11:36:02.485679+0000","last_fullsized":"2024-03-24T11:36:02.485679+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:57.745141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:57.745141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:18.976119+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"213'19","reported_seq":685,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.638713+0000","last_change":"2024-03-24T11:33:40.546128+0000","last_active":"2024-03-24T11:36:04.638713+0000","last_peered":"2024-03-24T11:36:04.638713+0000","last_clean":"2024-03-24T11:36:04.638713+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:36:04.638713+0000","last_undegraded":"2024-03-24T11:36:04.638713+0000","last_fullsized":"2024-03-24T11:36:04.638713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:33:40.545997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:40.545997+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":672,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488058+0000","last_change":"2024-03-24T11:31:40.668063+0000","last_active":"2024-03-24T11:36:02.488058+0000","last_peered":"2024-03-24T11:36:02.488058+0000","last_clean":"2024-03-24T11:36:02.488058+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:36:02.488058+0000","last_undegraded":"2024-03-24T11:36:02.488058+0000","last_fullsized":"2024-03-24T11:36:02.488058+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:40.667849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:31:40.667849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":280,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.490637+0000","last_change":"2024-03-24T11:32:23.732096+0000","last_active":"2024-03-24T11:36:06.490637+0000","last_peered":"2024-03-24T11:36:06.490637+0000","last_clean":"2024-03-24T11:36:06.490637+0000","last_became_active":"2024-03-24T11:30:35.353571+0000","last_became_peered":"2024-03-24T11:30:35.353571+0000","last_unstale":"2024-03-24T11:36:06.490637+0000","last_undegraded":"2024-03-24T11:36:06.490637+0000","last_fullsized":"2024-03-24T11:36:06.490637+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:23.731967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:23.731967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.622027+0000","last_change":"2024-03-24T11:30:24.313618+0000","last_active":"2024-03-24T11:36:04.622027+0000","last_peered":"2024-03-24T11:36:04.622027+0000","last_clean":"2024-03-24T11:36:04.622027+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:36:04.622027+0000","last_undegraded":"2024-03-24T11:36:04.622027+0000","last_fullsized":"2024-03-24T11:36:04.622027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:30:24.313152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:30:24.313152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"213'10","reported_seq":668,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.623790+0000","last_change":"2024-03-24T11:33:33.501770+0000","last_active":"2024-03-24T11:36:04.623790+0000","last_peered":"2024-03-24T11:36:04.623790+0000","last_clean":"2024-03-24T11:36:04.623790+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:36:04.623790+0000","last_undegraded":"2024-03-24T11:36:04.623790+0000","last_fullsized":"2024-03-24T11:36:04.623790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:33:33.501639+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.501639+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.994485+0000","last_change":"2024-03-24T11:36:02.994521+0000","last_active":"2024-03-24T11:36:02.994485+0000","last_peered":"2024-03-24T11:36:02.994485+0000","last_clean":"2024-03-24T11:36:02.994485+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:36:02.994485+0000","last_undegraded":"2024-03-24T11:36:02.994485+0000","last_fullsized":"2024-03-24T11:36:02.994485+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:02.994383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:36:02.994383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:32.550429+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478148+0000","last_change":"2024-03-24T11:32:38.991091+0000","last_active":"2024-03-24T11:36:02.478148+0000","last_peered":"2024-03-24T11:36:02.478148+0000","last_clean":"2024-03-24T11:36:02.478148+0000","last_became_active":"2024-03-24T11:18:57.700479+0000","last_became_peered":"2024-03-24T11:18:57.700479+0000","last_unstale":"2024-03-24T11:36:02.478148+0000","last_undegraded":"2024-03-24T11:36:02.478148+0000","last_fullsized":"2024-03-24T11:36:02.478148+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.991009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.991009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":284,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.482257+0000","last_change":"2024-03-24T11:32:43.213245+0000","last_active":"2024-03-24T11:36:02.482257+0000","last_peered":"2024-03-24T11:36:02.482257+0000","last_clean":"2024-03-24T11:36:02.482257+0000","last_became_active":"2024-03-24T11:30:35.353704+0000","last_became_peered":"2024-03-24T11:30:35.353704+0000","last_unstale":"2024-03-24T11:36:02.482257+0000","last_undegraded":"2024-03-24T11:36:02.482257+0000","last_fullsized":"2024-03-24T11:36:02.482257+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:43.213130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:32:43.213130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"365'8138","reported_seq":13366,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.525903+0000","last_change":"2024-03-24T11:35:44.651768+0000","last_active":"2024-03-24T11:36:03.525903+0000","last_peered":"2024-03-24T11:36:03.525903+0000","last_clean":"2024-03-24T11:36:03.525903+0000","last_became_active":"2024-03-24T11:35:43.803447+0000","last_became_peered":"2024-03-24T11:35:43.803447+0000","last_unstale":"2024-03-24T11:36:03.525903+0000","last_undegraded":"2024-03-24T11:36:03.525903+0000","last_fullsized":"2024-03-24T11:36:03.525903+0000","mapping_epoch":346,"log_start":"274'1821","ondisk_log_start":"274'1821","created":288,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"348'7608","last_scrub_stamp":"2024-03-24T11:35:44.651626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:44.651626+0000","objects_scrubbed":1550,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:07.643776+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66191360,"num_objects":1010,"num_object_clones":0,"num_object_copies":3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":540,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"213'24","reported_seq":721,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.479068+0000","last_change":"2024-03-24T11:32:30.897616+0000","last_active":"2024-03-24T11:36:02.479068+0000","last_peered":"2024-03-24T11:36:02.479068+0000","last_clean":"2024-03-24T11:36:02.479068+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:36:02.479068+0000","last_undegraded":"2024-03-24T11:36:02.479068+0000","last_fullsized":"2024-03-24T11:36:02.479068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:32:30.897405+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.897405+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":658,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.623699+0000","last_change":"2024-03-24T11:32:30.073961+0000","last_active":"2024-03-24T11:36:04.623699+0000","last_peered":"2024-03-24T11:36:04.623699+0000","last_clean":"2024-03-24T11:36:04.623699+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:36:04.623699+0000","last_undegraded":"2024-03-24T11:36:04.623699+0000","last_fullsized":"2024-03-24T11:36:04.623699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:30.073873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:32:30.073873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":665,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488631+0000","last_change":"2024-03-24T11:33:34.153970+0000","last_active":"2024-03-24T11:36:02.488631+0000","last_peered":"2024-03-24T11:36:02.488631+0000","last_clean":"2024-03-24T11:36:02.488631+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:36:02.488631+0000","last_undegraded":"2024-03-24T11:36:02.488631+0000","last_fullsized":"2024-03-24T11:36:02.488631+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:34.153751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:33:34.153751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":679,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.510530+0000","last_change":"2024-03-24T11:36:02.264753+0000","last_active":"2024-03-24T11:36:06.510530+0000","last_peered":"2024-03-24T11:36:06.510530+0000","last_clean":"2024-03-24T11:36:06.510530+0000","last_became_active":"2024-03-24T11:18:59.705380+0000","last_became_peered":"2024-03-24T11:18:59.705380+0000","last_unstale":"2024-03-24T11:36:06.510530+0000","last_undegraded":"2024-03-24T11:36:06.510530+0000","last_fullsized":"2024-03-24T11:36:06.510530+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:02.264489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:31:41.853751+0000","last_clean_scrub_stamp":"2024-03-24T11:36:02.264489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:06.895131+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"367'8262","reported_seq":13308,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.323390+0000","last_change":"2024-03-24T11:36:02.461837+0000","last_active":"2024-03-24T11:36:08.323390+0000","last_peered":"2024-03-24T11:36:08.323390+0000","last_clean":"2024-03-24T11:36:08.323390+0000","last_became_active":"2024-03-24T11:34:06.445898+0000","last_became_peered":"2024-03-24T11:34:06.445898+0000","last_unstale":"2024-03-24T11:36:08.323390+0000","last_undegraded":"2024-03-24T11:36:08.323390+0000","last_fullsized":"2024-03-24T11:36:08.323390+0000","mapping_epoch":294,"log_start":"281'3638","ondisk_log_start":"281'3638","created":288,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":5,"last_scrub":"364'8107","last_scrub_stamp":"2024-03-24T11:36:02.461562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:36:02.461562+0000","objects_scrubbed":1073,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:03.542207+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59965440,"num_objects":915,"num_object_clones":0,"num_object_copies":2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":158,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"213'20","reported_seq":725,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.488366+0000","last_change":"2024-03-24T11:33:41.275622+0000","last_active":"2024-03-24T11:36:02.488366+0000","last_peered":"2024-03-24T11:36:02.488366+0000","last_clean":"2024-03-24T11:36:02.488366+0000","last_became_active":"2024-03-24T11:33:33.699957+0000","last_became_peered":"2024-03-24T11:33:33.699957+0000","last_unstale":"2024-03-24T11:36:02.488366+0000","last_undegraded":"2024-03-24T11:36:02.488366+0000","last_fullsized":"2024-03-24T11:36:02.488366+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:41.275381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:41.275381+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":693,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.514011+0000","last_change":"2024-03-24T11:33:31.339856+0000","last_active":"2024-03-24T11:36:06.514011+0000","last_peered":"2024-03-24T11:36:06.514011+0000","last_clean":"2024-03-24T11:36:06.514011+0000","last_became_active":"2024-03-24T11:18:53.677439+0000","last_became_peered":"2024-03-24T11:18:53.677439+0000","last_unstale":"2024-03-24T11:36:06.514011+0000","last_undegraded":"2024-03-24T11:36:06.514011+0000","last_fullsized":"2024-03-24T11:36:06.514011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:33:31.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:31.339716+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"87'9","reported_seq":910,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478970+0000","last_change":"2024-03-24T11:35:56.347040+0000","last_active":"2024-03-24T11:36:02.478970+0000","last_peered":"2024-03-24T11:36:02.478970+0000","last_clean":"2024-03-24T11:36:02.478970+0000","last_became_active":"2024-03-24T11:18:57.701279+0000","last_became_peered":"2024-03-24T11:18:57.701279+0000","last_unstale":"2024-03-24T11:36:02.478970+0000","last_undegraded":"2024-03-24T11:36:02.478970+0000","last_fullsized":"2024-03-24T11:36:02.478970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'9","last_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:35:56.346906+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:09.971371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":323,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487377+0000","last_change":"2024-03-24T11:33:27.437600+0000","last_active":"2024-03-24T11:36:02.487377+0000","last_peered":"2024-03-24T11:36:02.487377+0000","last_clean":"2024-03-24T11:36:02.487377+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:36:02.487377+0000","last_undegraded":"2024-03-24T11:36:02.487377+0000","last_fullsized":"2024-03-24T11:36:02.487377+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:33:27.437477+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:33:27.437477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.13","version":"367'8540","reported_seq":16306,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.026085+0000","last_change":"2024-03-24T11:35:58.036995+0000","last_active":"2024-03-24T11:36:08.026085+0000","last_peered":"2024-03-24T11:36:08.026085+0000","last_clean":"2024-03-24T11:36:08.026085+0000","last_became_active":"2024-03-24T11:35:45.284529+0000","last_became_peered":"2024-03-24T11:35:45.284529+0000","last_unstale":"2024-03-24T11:36:08.026085+0000","last_undegraded":"2024-03-24T11:36:08.026085+0000","last_fullsized":"2024-03-24T11:36:08.026085+0000","mapping_epoch":348,"log_start":"276'2710","ondisk_log_start":"276'2710","created":288,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":5,"last_scrub":"361'8279","last_scrub_stamp":"2024-03-24T11:35:58.036817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:58.036817+0000","objects_scrubbed":1189,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:21.834112+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60817408,"num_objects":928,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":261,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"213'18","reported_seq":677,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.639326+0000","last_change":"2024-03-24T11:31:11.260142+0000","last_active":"2024-03-24T11:36:04.639326+0000","last_peered":"2024-03-24T11:36:04.639326+0000","last_clean":"2024-03-24T11:36:04.639326+0000","last_became_active":"2024-03-24T11:18:55.684389+0000","last_became_peered":"2024-03-24T11:18:55.684389+0000","last_unstale":"2024-03-24T11:36:04.639326+0000","last_undegraded":"2024-03-24T11:36:04.639326+0000","last_fullsized":"2024-03-24T11:36:04.639326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:31:11.259991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:11.259991+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":667,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491175+0000","last_change":"2024-03-24T11:33:19.752899+0000","last_active":"2024-03-24T11:36:06.491175+0000","last_peered":"2024-03-24T11:36:06.491175+0000","last_clean":"2024-03-24T11:36:06.491175+0000","last_became_active":"2024-03-24T11:18:53.672100+0000","last_became_peered":"2024-03-24T11:18:53.672100+0000","last_unstale":"2024-03-24T11:36:06.491175+0000","last_undegraded":"2024-03-24T11:36:06.491175+0000","last_fullsized":"2024-03-24T11:36:06.491175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.752717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.752717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":667,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.492095+0000","last_change":"2024-03-24T11:32:38.742586+0000","last_active":"2024-03-24T11:36:06.492095+0000","last_peered":"2024-03-24T11:36:06.492095+0000","last_clean":"2024-03-24T11:36:06.492095+0000","last_became_active":"2024-03-24T11:18:57.699098+0000","last_became_peered":"2024-03-24T11:18:57.699098+0000","last_unstale":"2024-03-24T11:36:06.492095+0000","last_undegraded":"2024-03-24T11:36:06.492095+0000","last_fullsized":"2024-03-24T11:36:06.492095+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:38.742478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:38.742478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":650,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.640048+0000","last_change":"2024-03-24T11:35:49.842222+0000","last_active":"2024-03-24T11:36:04.640048+0000","last_peered":"2024-03-24T11:36:04.640048+0000","last_clean":"2024-03-24T11:36:04.640048+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:36:04.640048+0000","last_undegraded":"2024-03-24T11:36:04.640048+0000","last_fullsized":"2024-03-24T11:36:04.640048+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:49.842117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:49.842117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:54.344422+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"365'8323","reported_seq":18560,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.435857+0000","last_change":"2024-03-24T11:35:42.037947+0000","last_active":"2024-03-24T11:36:03.435857+0000","last_peered":"2024-03-24T11:36:03.435857+0000","last_clean":"2024-03-24T11:36:03.435857+0000","last_became_active":"2024-03-24T11:35:38.740694+0000","last_became_peered":"2024-03-24T11:35:38.740694+0000","last_unstale":"2024-03-24T11:36:03.435857+0000","last_undegraded":"2024-03-24T11:36:03.435857+0000","last_fullsized":"2024-03-24T11:36:03.435857+0000","mapping_epoch":341,"log_start":"274'1727","ondisk_log_start":"274'1727","created":288,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'7652","last_scrub_stamp":"2024-03-24T11:35:42.037604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:42.037604+0000","objects_scrubbed":1724,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:03.979950+0000","scrub_duration":569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68485120,"num_objects":1045,"num_object_clones":0,"num_object_copies":3135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":679,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"213'27","reported_seq":726,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.478914+0000","last_change":"2024-03-24T11:33:40.976361+0000","last_active":"2024-03-24T11:36:02.478914+0000","last_peered":"2024-03-24T11:36:02.478914+0000","last_clean":"2024-03-24T11:36:02.478914+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:36:02.478914+0000","last_undegraded":"2024-03-24T11:36:02.478914+0000","last_fullsized":"2024-03-24T11:36:02.478914+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:33:40.976211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:40.976211+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":671,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.489874+0000","last_change":"2024-03-24T11:33:22.182976+0000","last_active":"2024-03-24T11:36:02.489874+0000","last_peered":"2024-03-24T11:36:02.489874+0000","last_clean":"2024-03-24T11:36:02.489874+0000","last_became_active":"2024-03-24T11:18:53.681560+0000","last_became_peered":"2024-03-24T11:18:53.681560+0000","last_unstale":"2024-03-24T11:36:02.489874+0000","last_undegraded":"2024-03-24T11:36:02.489874+0000","last_fullsized":"2024-03-24T11:36:02.489874+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:22.182859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:22.182859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":675,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.510377+0000","last_change":"2024-03-24T11:32:42.808822+0000","last_active":"2024-03-24T11:36:06.510377+0000","last_peered":"2024-03-24T11:36:06.510377+0000","last_clean":"2024-03-24T11:36:06.510377+0000","last_became_active":"2024-03-24T11:18:57.698225+0000","last_became_peered":"2024-03-24T11:18:57.698225+0000","last_unstale":"2024-03-24T11:36:06.510377+0000","last_undegraded":"2024-03-24T11:36:06.510377+0000","last_fullsized":"2024-03-24T11:36:06.510377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:42.808738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:32:42.808738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":665,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491834+0000","last_change":"2024-03-24T11:33:42.035003+0000","last_active":"2024-03-24T11:36:06.491834+0000","last_peered":"2024-03-24T11:36:06.491834+0000","last_clean":"2024-03-24T11:36:06.491834+0000","last_became_active":"2024-03-24T11:19:00.311768+0000","last_became_peered":"2024-03-24T11:19:00.311768+0000","last_unstale":"2024-03-24T11:36:06.491834+0000","last_undegraded":"2024-03-24T11:36:06.491834+0000","last_fullsized":"2024-03-24T11:36:06.491834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.034893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.034893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"366'9655","reported_seq":17290,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.857260+0000","last_change":"2024-03-24T11:35:56.947074+0000","last_active":"2024-03-24T11:36:04.857260+0000","last_peered":"2024-03-24T11:36:04.857260+0000","last_clean":"2024-03-24T11:36:04.857260+0000","last_became_active":"2024-03-24T11:34:48.528319+0000","last_became_peered":"2024-03-24T11:34:48.528319+0000","last_unstale":"2024-03-24T11:36:04.857260+0000","last_undegraded":"2024-03-24T11:36:04.857260+0000","last_fullsized":"2024-03-24T11:36:04.857260+0000","mapping_epoch":304,"log_start":"295'5488","ondisk_log_start":"295'5488","created":288,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'9431","last_scrub_stamp":"2024-03-24T11:35:56.946608+0000","last_deep_scrub":"360'9431","last_deep_scrub_stamp":"2024-03-24T11:35:56.946608+0000","last_clean_scrub_stamp":"2024-03-24T11:35:56.946608+0000","objects_scrubbed":1332,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:12.490075+0000","scrub_duration":8315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":349,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"213'7","reported_seq":684,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491222+0000","last_change":"2024-03-24T11:36:04.265825+0000","last_active":"2024-03-24T11:36:06.491222+0000","last_peered":"2024-03-24T11:36:06.491222+0000","last_clean":"2024-03-24T11:36:06.491222+0000","last_became_active":"2024-03-24T11:18:55.686128+0000","last_became_peered":"2024-03-24T11:18:55.686128+0000","last_unstale":"2024-03-24T11:36:06.491222+0000","last_undegraded":"2024-03-24T11:36:06.491222+0000","last_fullsized":"2024-03-24T11:36:06.491222+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:36:04.265719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:04.265719+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:30.526072+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":671,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487660+0000","last_change":"2024-03-24T11:33:19.374929+0000","last_active":"2024-03-24T11:36:02.487660+0000","last_peered":"2024-03-24T11:36:02.487660+0000","last_clean":"2024-03-24T11:36:02.487660+0000","last_became_active":"2024-03-24T11:18:53.681378+0000","last_became_peered":"2024-03-24T11:18:53.681378+0000","last_unstale":"2024-03-24T11:36:02.487660+0000","last_undegraded":"2024-03-24T11:36:02.487660+0000","last_fullsized":"2024-03-24T11:36:02.487660+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:19.374805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.374805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":280,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491290+0000","last_change":"2024-03-24T11:31:41.715026+0000","last_active":"2024-03-24T11:36:06.491290+0000","last_peered":"2024-03-24T11:36:06.491290+0000","last_clean":"2024-03-24T11:36:06.491290+0000","last_became_active":"2024-03-24T11:30:35.359239+0000","last_became_peered":"2024-03-24T11:30:35.359239+0000","last_unstale":"2024-03-24T11:36:06.491290+0000","last_undegraded":"2024-03-24T11:36:06.491290+0000","last_fullsized":"2024-03-24T11:36:06.491290+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.714877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.714877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.486646+0000","last_change":"2024-03-24T11:33:28.608037+0000","last_active":"2024-03-24T11:36:02.486646+0000","last_peered":"2024-03-24T11:36:02.486646+0000","last_clean":"2024-03-24T11:36:02.486646+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:36:02.486646+0000","last_undegraded":"2024-03-24T11:36:02.486646+0000","last_fullsized":"2024-03-24T11:36:02.486646+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:33:28.607919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"367'8402","reported_seq":15994,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.073223+0000","last_change":"2024-03-24T11:36:00.270987+0000","last_active":"2024-03-24T11:36:08.073223+0000","last_peered":"2024-03-24T11:36:08.073223+0000","last_clean":"2024-03-24T11:36:08.073223+0000","last_became_active":"2024-03-24T11:35:58.457252+0000","last_became_peered":"2024-03-24T11:35:58.457252+0000","last_unstale":"2024-03-24T11:36:08.073223+0000","last_undegraded":"2024-03-24T11:36:08.073223+0000","last_fullsized":"2024-03-24T11:36:08.073223+0000","mapping_epoch":361,"log_start":"276'2529","ondisk_log_start":"276'2529","created":288,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":5,"last_scrub":"363'8176","last_scrub_stamp":"2024-03-24T11:36:00.270673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.270673+0000","objects_scrubbed":1107,"log_size":5873,"log_dups_size":0,"ondisk_log_size":5873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:25.203229+0000","scrub_duration":360,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57409536,"num_objects":876,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"213'15","reported_seq":688,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.491091+0000","last_change":"2024-03-24T11:33:24.846253+0000","last_active":"2024-03-24T11:36:06.491091+0000","last_peered":"2024-03-24T11:36:06.491091+0000","last_clean":"2024-03-24T11:36:06.491091+0000","last_became_active":"2024-03-24T11:18:55.681483+0000","last_became_peered":"2024-03-24T11:18:55.681483+0000","last_unstale":"2024-03-24T11:36:06.491091+0000","last_undegraded":"2024-03-24T11:36:06.491091+0000","last_fullsized":"2024-03-24T11:36:06.491091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:33:24.845898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.845898+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":671,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487828+0000","last_change":"2024-03-24T11:31:41.687584+0000","last_active":"2024-03-24T11:36:02.487828+0000","last_peered":"2024-03-24T11:36:02.487828+0000","last_clean":"2024-03-24T11:36:02.487828+0000","last_became_active":"2024-03-24T11:18:53.680344+0000","last_became_peered":"2024-03-24T11:18:53.680344+0000","last_unstale":"2024-03-24T11:36:02.487828+0000","last_undegraded":"2024-03-24T11:36:02.487828+0000","last_fullsized":"2024-03-24T11:36:02.487828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:41.687474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.687474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":671,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487172+0000","last_change":"2024-03-24T11:33:38.555137+0000","last_active":"2024-03-24T11:36:02.487172+0000","last_peered":"2024-03-24T11:36:02.487172+0000","last_clean":"2024-03-24T11:36:02.487172+0000","last_became_active":"2024-03-24T11:18:57.694173+0000","last_became_peered":"2024-03-24T11:18:57.694173+0000","last_unstale":"2024-03-24T11:36:02.487172+0000","last_undegraded":"2024-03-24T11:36:02.487172+0000","last_fullsized":"2024-03-24T11:36:02.487172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:38.554887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.554887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.486028+0000","last_change":"2024-03-24T11:33:24.183098+0000","last_active":"2024-03-24T11:36:02.486028+0000","last_peered":"2024-03-24T11:36:02.486028+0000","last_clean":"2024-03-24T11:36:02.486028+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:36:02.486028+0000","last_undegraded":"2024-03-24T11:36:02.486028+0000","last_fullsized":"2024-03-24T11:36:02.486028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:24.182966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.182966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"367'8289","reported_seq":13314,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:08.320251+0000","last_change":"2024-03-24T11:36:03.490067+0000","last_active":"2024-03-24T11:36:08.320251+0000","last_peered":"2024-03-24T11:36:08.320251+0000","last_clean":"2024-03-24T11:36:08.320251+0000","last_became_active":"2024-03-24T11:34:30.364787+0000","last_became_peered":"2024-03-24T11:34:30.364787+0000","last_unstale":"2024-03-24T11:36:08.320251+0000","last_undegraded":"2024-03-24T11:36:08.320251+0000","last_fullsized":"2024-03-24T11:36:08.320251+0000","mapping_epoch":297,"log_start":"281'3634","ondisk_log_start":"281'3634","created":288,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":5,"last_scrub":"365'8163","last_scrub_stamp":"2024-03-24T11:36:03.489895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:36:03.489895+0000","objects_scrubbed":1024,"log_size":4655,"log_dups_size":0,"ondisk_log_size":4655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:11.847948+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58785792,"num_objects":897,"num_object_clones":0,"num_object_copies":2691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"213'27","reported_seq":689,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.638808+0000","last_change":"2024-03-24T11:31:41.107651+0000","last_active":"2024-03-24T11:36:04.638808+0000","last_peered":"2024-03-24T11:36:04.638808+0000","last_clean":"2024-03-24T11:36:04.638808+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:36:04.638808+0000","last_undegraded":"2024-03-24T11:36:04.638808+0000","last_fullsized":"2024-03-24T11:36:04.638808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:31:41.107512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:31:41.107512+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":659,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.639087+0000","last_change":"2024-03-24T11:35:51.761976+0000","last_active":"2024-03-24T11:36:04.639087+0000","last_peered":"2024-03-24T11:36:04.639087+0000","last_clean":"2024-03-24T11:36:04.639087+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:36:04.639087+0000","last_undegraded":"2024-03-24T11:36:04.639087+0000","last_fullsized":"2024-03-24T11:36:04.639087+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:51.761850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:51.761850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:19.998874+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":660,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.490706+0000","last_change":"2024-03-24T11:32:24.692923+0000","last_active":"2024-03-24T11:36:06.490706+0000","last_peered":"2024-03-24T11:36:06.490706+0000","last_clean":"2024-03-24T11:36:06.490706+0000","last_became_active":"2024-03-24T11:18:57.698079+0000","last_became_peered":"2024-03-24T11:18:57.698079+0000","last_unstale":"2024-03-24T11:36:06.490706+0000","last_undegraded":"2024-03-24T11:36:06.490706+0000","last_fullsized":"2024-03-24T11:36:06.490706+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:24.692844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:24.692844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":637,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-24T11:36:04.646029+0000","last_change":"2024-03-24T11:31:14.327893+0000","last_active":"2024-03-24T11:36:04.646029+0000","last_peered":"2024-03-24T11:36:04.646029+0000","last_clean":"2024-03-24T11:36:04.646029+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:36:04.646029+0000","last_undegraded":"2024-03-24T11:36:04.646029+0000","last_fullsized":"2024-03-24T11:36:04.646029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:31:14.327777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:31:14.327777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"365'8265","reported_seq":15661,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:03.411090+0000","last_change":"2024-03-24T11:35:41.023492+0000","last_active":"2024-03-24T11:36:03.411090+0000","last_peered":"2024-03-24T11:36:03.411090+0000","last_clean":"2024-03-24T11:36:03.411090+0000","last_became_active":"2024-03-24T11:34:58.280491+0000","last_became_peered":"2024-03-24T11:34:58.280491+0000","last_unstale":"2024-03-24T11:36:03.411090+0000","last_undegraded":"2024-03-24T11:36:03.411090+0000","last_fullsized":"2024-03-24T11:36:03.411090+0000","mapping_epoch":309,"log_start":"274'1333","ondisk_log_start":"274'1333","created":288,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":5,"last_scrub":"344'7609","last_scrub_stamp":"2024-03-24T11:35:41.023108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:51.307286+0000","last_clean_scrub_stamp":"2024-03-24T11:35:41.023108+0000","objects_scrubbed":1705,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:36:53.239062+0000","scrub_duration":512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68222976,"num_objects":1041,"num_object_clones":0,"num_object_copies":3123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"213'15","reported_seq":713,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.490695+0000","last_change":"2024-03-24T11:33:38.953403+0000","last_active":"2024-03-24T11:36:06.490695+0000","last_peered":"2024-03-24T11:36:06.490695+0000","last_clean":"2024-03-24T11:36:06.490695+0000","last_became_active":"2024-03-24T11:19:18.816615+0000","last_became_peered":"2024-03-24T11:19:18.816615+0000","last_unstale":"2024-03-24T11:36:06.490695+0000","last_undegraded":"2024-03-24T11:36:06.490695+0000","last_fullsized":"2024-03-24T11:36:06.490695+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:33:38.953051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.953051+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":673,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-24T11:36:02.487719+0000","last_change":"2024-03-24T11:36:01.688860+0000","last_active":"2024-03-24T11:36:02.487719+0000","last_peered":"2024-03-24T11:36:02.487719+0000","last_clean":"2024-03-24T11:36:02.487719+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:36:02.487719+0000","last_undegraded":"2024-03-24T11:36:02.487719+0000","last_fullsized":"2024-03-24T11:36:02.487719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:01.688732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:36:01.688732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:37:03.130114+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":676,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.513877+0000","last_change":"2024-03-24T11:32:28.712422+0000","last_active":"2024-03-24T11:36:06.513877+0000","last_peered":"2024-03-24T11:36:06.513877+0000","last_clean":"2024-03-24T11:36:06.513877+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:36:06.513877+0000","last_undegraded":"2024-03-24T11:36:06.513877+0000","last_fullsized":"2024-03-24T11:36:06.513877+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:32:28.712294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:32:28.712294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":663,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-03-24T11:36:06.490844+0000","last_change":"2024-03-24T11:33:36.020005+0000","last_active":"2024-03-24T11:36:06.490844+0000","last_peered":"2024-03-24T11:36:06.490844+0000","last_clean":"2024-03-24T11:36:06.490844+0000","last_became_active":"2024-03-24T11:18:59.704034+0000","last_became_peered":"2024-03-24T11:18:59.704034+0000","last_unstale":"2024-03-24T11:36:06.490844+0000","last_undegraded":"2024-03-24T11:36:06.490844+0000","last_fullsized":"2024-03-24T11:36:06.490844+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:36.019890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:36.019890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":2054553624,"num_objects":31351,"num_object_clones":0,"num_object_copies":94053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31351,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90019,"num_write_kb":3098753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2475,"num_bytes_recovered":162136064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6511931392,"data_stored":6511919176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150825,"ondisk_log_size":150825,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":338,"num_read_kb":333,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1126,"num_read_kb":883,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":737,"ondisk_log_size":737,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070675,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080900,"kb_used_data":486072,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92664252,"statfs":{"total":95995035648,"available":94888194048,"internally_reserved":0,"allocated":497737728,"data_stored":497297752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":609089995},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365015,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511332,"kb_used_data":1047000,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92233820,"statfs":{"total":95995035648,"available":94447431680,"internally_reserved":0,"allocated":1072128000,"data_stored":1071704595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":475462086},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575090,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1245852,"kb_used_data":574676,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92499300,"statfs":{"total":95995035648,"available":94719283200,"internally_reserved":0,"allocated":588468224,"data_stored":588000634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738614,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1631224,"kb_used_data":1000300,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92113928,"statfs":{"total":95995035648,"available":94324662272,"internally_reserved":0,"allocated":1024307200,"data_stored":1023862849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1464752,"kb_used_data":905952,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92280400,"statfs":{"total":95995035648,"available":94495129600,"internally_reserved":0,"allocated":927694848,"data_stored":927263066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149925,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1533268,"kb_used_data":902280,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92211884,"statfs":{"total":95995035648,"available":94424969216,"internally_reserved":0,"allocated":923934720,"data_stored":923470958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":646052295},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986402,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1409296,"kb_used_data":864392,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92335856,"statfs":{"total":95995035648,"available":94551916544,"internally_reserved":0,"allocated":885137408,"data_stored":884681500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659356,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301416,"kb_used_data":670492,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92443736,"statfs":{"total":95995035648,"available":94662385664,"internally_reserved":0,"allocated":686583808,"data_stored":686156388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1003425792,"data_stored":1003421720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":581566464,"data_stored":581566464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":903282688,"data_stored":903282688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":918880256,"data_stored":918880256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":880611328,"data_stored":880607256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":677249024,"data_stored":677249024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1069158400,"data_stored":1069154328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":477757440,"data_stored":477757440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:36:11.197 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T11:36:11.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 36 2024-03-24T11:36:11.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:11.338 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:11.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:11.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:11.482 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:11.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:11.583 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:11.621 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:11.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:11.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:11.761 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:11.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:11.798 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:11 smithi179 ceph-mon[29608]: pgmap v786: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-24T11:36:11.798 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:11 smithi179 ceph-mon[37907]: pgmap v786: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-24T11:36:11.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:11.894 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:11.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:11.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:11 smithi181 ceph-mon[33998]: pgmap v786: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-24T11:36:11.914 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:11.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:36:11.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:12.032 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:12.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:12.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:12.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:12.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:12.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:12.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:12.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:12.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:12.452 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:12.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:12.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:12.583 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:12.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:12.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:12.723 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:12.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:12.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:12.862 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:12.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:12.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:12.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:13.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:13.137 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:13.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.266 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:13.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:13.367 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:13.402 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:13.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:13.541 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:13.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:13.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:13.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:13.807 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:13.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:36:13.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:13 smithi179 ceph-mon[29608]: pgmap v787: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:36:13.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:13 smithi179 ceph-mon[37907]: pgmap v787: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:36:13.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:13 smithi181 ceph-mon[33998]: pgmap v787: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:36:13.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:13.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:13.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:14.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:14.090 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:14.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:14.192 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:14.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:14.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:14.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:14.373 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:14.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:14.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:36:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:36:14.475 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:14.515 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:14.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:14.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:14.648 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:14.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:14.750 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:14.786 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:14.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:14.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:14.924 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:14.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:15.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:15.053 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:15.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:15.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:15.163 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:15.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:15.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:15.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:15.300 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:15.300 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:15 smithi179 ceph-mon[29608]: pgmap v788: 168 pgs: 168 active+clean; 1.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:15.300 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:36:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:36:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:36:15.300 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:15.300 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:15 smithi179 ceph-mon[37907]: pgmap v788: 168 pgs: 168 active+clean; 1.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:15.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:15.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:15.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:15.476 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:15.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:15.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:15.612 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:15.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:15.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:15.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:15 smithi181 ceph-mon[33998]: pgmap v788: 168 pgs: 168 active+clean; 1.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:15.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:15.743 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:15.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:15.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:15.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:15.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:15.981 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:16.022 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:16.124 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.159 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:16.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:16.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:16.293 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:16.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:16.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:16.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:16.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:16.582 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:16.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:16.718 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:16.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:16.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:16.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:16.996 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:16.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:17.052 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:17.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:17.143 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:17.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:17.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:17.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:17.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:17.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:17.419 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:17.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:17.520 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:17.554 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:17.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:17.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:17.690 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:17.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:17.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:17 smithi179 ceph-mon[29608]: pgmap v789: 168 pgs: 168 active+clean; 1.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:36:17.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:17 smithi179 ceph-mon[37907]: pgmap v789: 168 pgs: 168 active+clean; 1.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:36:17.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:17.827 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:17.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:17 smithi181 ceph-mon[33998]: pgmap v789: 168 pgs: 168 active+clean; 1.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:36:17.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:17.967 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:17.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:18.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:18.104 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:18.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:18.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:18.248 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:18.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:18.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:18.387 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:18.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:18.489 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:18.524 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:18.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:18.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:18.665 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:18.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:18.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:18.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:18.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:18.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:18.945 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:18.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:19.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:19.083 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:19.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:19.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:19.218 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:19.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:19.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:19.358 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:19.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:19.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:19.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:19.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:19.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:19.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:19.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:19.726 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:19 smithi179 ceph-mon[29608]: pgmap v790: 168 pgs: 168 active+clean; 1.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:36:19.726 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:19.726 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:19.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:19 smithi179 ceph-mon[37907]: pgmap v790: 168 pgs: 168 active+clean; 1.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:36:19.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:19.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:19.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:19.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:19.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:19.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:19.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:19 smithi181 ceph-mon[33998]: pgmap v790: 168 pgs: 168 active+clean; 1.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:36:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:36:19.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:19.927 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:19.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:20.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:20.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:20.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:20.162 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:20.198 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:20.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:20.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:20.335 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:20.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:20.436 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:20.469 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:20.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:20.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:20.610 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:20.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:20.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:20.752 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:20.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:20.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:20.891 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:20.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:20.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:21.032 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:21.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:21.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:21.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:21.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:21.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:21.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:21.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:21.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:21.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:21.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:21.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:21.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:21.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:21.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:21.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:21.712 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:36:21.746 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:36:21.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:21.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:21.798 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:36:21.850 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:36:21.850 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:21 smithi179 ceph-mon[29608]: pgmap v791: 168 pgs: 168 active+clean; 978 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 307 op/s 2024-03-24T11:36:21.850 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:21 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1971472165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:21.850 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:21 smithi179 ceph-mon[37907]: pgmap v791: 168 pgs: 168 active+clean; 978 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 307 op/s 2024-03-24T11:36:21.850 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:21 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1971472165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:21.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:21.894 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:21.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:21.904 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:36:21.944 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:36:21.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:22.009 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:36:22.039 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:22.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:22.049 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:36:22.064 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:36:22.064 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:36:22.064 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:36:22.064 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:36:22.064 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:36:22.064 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:36:22.065 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:36:22.065 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:36:22.074 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:36:22.074 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.15621.0:209933 10.f e367/267 10:f4735b9c:::benchmark_data_smithi179_155357_object99437:head v 367'13019, mlcod=367'13017)", 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:22.070541+0000", 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.0035758309999999998, 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036107700000000001, 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070541+0000", 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:22.075 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070541+0000", 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070542+0000", 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0020000000000001e-06 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070553+0000", 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1522000000000001e-05 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:22.076 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070571+0000", 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7900000000000001e-05 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070580+0000", 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1050000000000001e-06 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070744+0000", 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016377899999999999 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:36:22.077 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.070797+0000", 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.3337000000000001e-05 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.074139+0000", 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003341428 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_repop(client.15621.0:209941 10.f e367/267)", 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:22.072610+0000", 2024-03-24T11:36:22.078 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 0.001506952, 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0016153109999999999, 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "queued for pg", 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.072610+0000", 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.072610+0000", 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:36:22.079 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.072611+0000", 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.71e-07 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.072621+0000", 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0037000000000001e-05 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.072637+0000", 2024-03-24T11:36:22.080 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6222000000000002e-05 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:22.072646+0000", 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7409999999999998e-06 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: ], 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout: "num_ops": 2 2024-03-24T11:36:22.081 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:36:22.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:22.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:21 smithi181 ceph-mon[33998]: pgmap v791: 168 pgs: 168 active+clean; 978 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 307 op/s 2024-03-24T11:36:22.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:21 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1971472165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:22.176 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:22.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:22.278 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:22.323 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:22.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:22.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:22.461 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:22.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:22.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:22.600 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:22.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:22.681 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 10 pg_num to 36 2024-03-24T11:36:22.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:22.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:22.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:22.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:22.879 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:22.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:22 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2190045494' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-24T11:36:22.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:22 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-24T11:36:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:22 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2190045494' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-24T11:36:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:22 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-24T11:36:22.981 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:23.018 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:23.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:22 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2190045494' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-24T11:36:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:22 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]: dispatch 2024-03-24T11:36:23.153 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:23.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:23.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:23.391 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:23.430 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:23.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:23.560 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:23.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:23.685 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:23.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:23.823 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:23.924 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:23 smithi179 ceph-mon[29608]: pgmap v792: 168 pgs: 168 active+clean; 978 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 568 op/s 2024-03-24T11:36:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-03-24T11:36:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:23 smithi179 ceph-mon[29608]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T11:36:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:23 smithi179 ceph-mon[37907]: pgmap v792: 168 pgs: 168 active+clean; 978 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 568 op/s 2024-03-24T11:36:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-03-24T11:36:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:23 smithi179 ceph-mon[37907]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T11:36:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:23.974 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:23.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:24.066 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:23 smithi181 ceph-mon[33998]: pgmap v792: 168 pgs: 168 active+clean; 978 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 568 op/s 2024-03-24T11:36:24.066 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "36"}]': finished 2024-03-24T11:36:24.066 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:23 smithi181 ceph-mon[33998]: osdmap e368: 8 total, 8 up, 8 in 2024-03-24T11:36:24.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:24.110 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:24.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:24.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:24.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:24.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:24.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:24.391 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:24.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:24.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:36:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:36:24.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:24.520 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:24.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:24.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:24.666 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:24.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:24.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:24.806 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:24.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:24.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:24.949 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:24.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:25.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:25.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:25.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:25.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:25.221 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:25.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:25.297 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:36:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:36:25.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:25.376 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:25.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:25.478 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:25.516 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:25.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:25.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:25.642 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:25.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:25 smithi181 ceph-mon[33998]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T11:36:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:25 smithi181 ceph-mon[33998]: pgmap v795: 168 pgs: 168 active+clean; 587 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:36:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:25 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:25 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[29608]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[29608]: pgmap v795: 168 pgs: 168 active+clean; 587 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[37907]: osdmap e369: 8 total, 8 up, 8 in 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[37907]: pgmap v795: 168 pgs: 168 active+clean; 587 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:25.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:25 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:25.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:25.767 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:25.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:25.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:25.891 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:25.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:25.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:26.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:26.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:26.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:26.250 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:26.275 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:26.320 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:26 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-03-24T11:36:26.320 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:26 smithi179 ceph-mon[29608]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T11:36:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:26 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-03-24T11:36:26.322 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:26 smithi179 ceph-mon[37907]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T11:36:26.362 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:36:26.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:26.407 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:26.508 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.549 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:26.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:26.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:26.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:26 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "36"}]': finished 2024-03-24T11:36:26.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:26 smithi181 ceph-mon[33998]: osdmap e370: 8 total, 8 up, 8 in 2024-03-24T11:36:26.673 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:26.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:26.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:26.808 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:26.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:26.948 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:26.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:27.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:27.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:27.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:27.225 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:27.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:27.363 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:27.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.489 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:27.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:27.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:27.613 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:27.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:27.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:27 smithi181 ceph-mon[33998]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T11:36:27.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:27 smithi181 ceph-mon[33998]: pgmap v798: 178 pgs: 10 unknown, 168 active+clean; 575 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T11:36:27.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:27 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:27.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:27 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[29608]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T11:36:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[29608]: pgmap v798: 178 pgs: 10 unknown, 168 active+clean; 575 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T11:36:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:27.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[37907]: osdmap e371: 8 total, 8 up, 8 in 2024-03-24T11:36:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[37907]: pgmap v798: 178 pgs: 10 unknown, 168 active+clean; 575 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-24T11:36:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:27.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:27 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-03-24T11:36:27.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:27.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:36:27.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:27.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:27.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:27.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:27.990 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:27.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:28.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:28.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:28.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:28.217 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:28.240 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:28.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:28.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:28.369 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:28.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:28.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:36:28.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:36:28.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-03-24T11:36:28.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[29608]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T11:36:28.419 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[29608]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T11:36:28.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:36:28.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:36:28.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-03-24T11:36:28.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[37907]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T11:36:28.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:28 smithi179 ceph-mon[37907]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T11:36:28.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:28.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:28.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:28.598 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:28.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:28 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:36:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:28 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:36:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:28 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "36"}]': finished 2024-03-24T11:36:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:28 smithi181 ceph-mon[33998]: osdmap e372: 8 total, 8 up, 8 in 2024-03-24T11:36:28.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:28 smithi181 ceph-mon[33998]: osdmap e373: 8 total, 8 up, 8 in 2024-03-24T11:36:28.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:28.750 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:28.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:28.760 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:28.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:28.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:28.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:28.976 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:28.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:29.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:29.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:29.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:29.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:29.266 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:29.341 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[29608]: pgmap v801: 178 pgs: 3 remapped+peering, 7 unknown, 168 active+clean; 479 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 265 op/s 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3988529831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[29608]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[37907]: pgmap v801: 178 pgs: 3 remapped+peering, 7 unknown, 168 active+clean; 479 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 265 op/s 2024-03-24T11:36:29.368 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3988529831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:29.369 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:29.369 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:29.369 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:36:29.369 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:29 smithi179 ceph-mon[37907]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T11:36:29.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:29.431 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:29.533 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:29.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:29 smithi181 ceph-mon[33998]: pgmap v801: 178 pgs: 3 remapped+peering, 7 unknown, 168 active+clean; 479 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 265 op/s 2024-03-24T11:36:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:29 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3988529831' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:29 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:29 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:29 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:36:29.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:29 smithi181 ceph-mon[33998]: osdmap e374: 8 total, 8 up, 8 in 2024-03-24T11:36:29.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:29.711 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:29.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:29.848 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:29.950 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:29.990 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:29.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:30.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:30.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:30.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:30.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:30.264 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:30.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:30.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:30.403 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:30.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:30.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:30.543 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:30.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:30.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:30.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 371'13302 2024-03-24T11:36:30.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 371'13408 2024-03-24T11:36:30.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.23 starting backfill to osd.4 from (0'0,0'0] MAX to 371'10370 2024-03-24T11:36:30.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 371'9031 2024-03-24T11:36:30.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 371'13511 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 371'13451 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 371'13302 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 371'8860 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 371'10370 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 371'8992 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 371'13451 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 371'13511 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 371'8992 2024-03-24T11:36:30.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:30 smithi181 ceph-mon[33998]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 371'8860 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 371'13302 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 371'13408 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.23 starting backfill to osd.4 from (0'0,0'0] MAX to 371'10370 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 371'9031 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 371'13511 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 371'13451 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 371'13302 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 371'8860 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 371'10370 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 371'8992 2024-03-24T11:36:30.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 371'13451 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 371'13511 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 371'8992 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[29608]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 371'8860 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1f starting backfill to osd.1 from (0'0,0'0] MAX to 371'13302 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1d starting backfill to osd.1 from (0'0,0'0] MAX to 371'13408 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1b starting backfill to osd.3 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.23 starting backfill to osd.4 from (0'0,0'0] MAX to 371'10370 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.21 starting backfill to osd.2 from (0'0,0'0] MAX to 371'9031 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1e starting backfill to osd.0 from (0'0,0'0] MAX to 371'13511 2024-03-24T11:36:30.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1c starting backfill to osd.1 from (0'0,0'0] MAX to 371'13451 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1f starting backfill to osd.7 from (0'0,0'0] MAX to 371'13302 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.20 starting backfill to osd.0 from (0'0,0'0] MAX to 371'8860 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.23 starting backfill to osd.7 from (0'0,0'0] MAX to 371'10370 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.22 starting backfill to osd.5 from (0'0,0'0] MAX to 371'8992 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1b starting backfill to osd.4 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1c starting backfill to osd.6 from (0'0,0'0] MAX to 371'13451 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1e starting backfill to osd.5 from (0'0,0'0] MAX to 371'13511 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.1b starting backfill to osd.7 from (0'0,0'0] MAX to 371'13137 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.22 starting backfill to osd.7 from (0'0,0'0] MAX to 371'8992 2024-03-24T11:36:30.656 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:30 smithi179 ceph-mon[37907]: 10.20 starting backfill to osd.2 from (0'0,0'0] MAX to 371'8860 2024-03-24T11:36:30.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:30.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:30.780 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:30.814 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:30.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:30.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:30.945 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:30.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:31.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:31.071 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:31.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:31.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:31.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:31.322 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:31.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:31.459 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:31.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.588 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:31.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:31.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:31 smithi181 ceph-mon[33998]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T11:36:31.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:31 smithi181 ceph-mon[33998]: pgmap v804: 178 pgs: 7 remapped+peering, 2 unknown, 169 active+clean; 332 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 604 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:36:31.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:31 smithi179 ceph-mon[29608]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T11:36:31.669 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:31 smithi179 ceph-mon[29608]: pgmap v804: 178 pgs: 7 remapped+peering, 2 unknown, 169 active+clean; 332 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 604 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:36:31.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:31 smithi179 ceph-mon[37907]: osdmap e375: 8 total, 8 up, 8 in 2024-03-24T11:36:31.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:31 smithi179 ceph-mon[37907]: pgmap v804: 178 pgs: 7 remapped+peering, 2 unknown, 169 active+clean; 332 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 604 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:36:31.689 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.714 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:31.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:31.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:31.839 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:31.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:31.964 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:31.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:32.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:32.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:32.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:32.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:32.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:32.216 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:32.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:32.317 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:32.342 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:32.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:32.444 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:32.481 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:32.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:32.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:36:32.583 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:32.638 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:32.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:32.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:32 smithi181 ceph-mon[33998]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T11:36:32.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:32 smithi181 ceph-mon[33998]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T11:36:32.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:32 smithi179 ceph-mon[37907]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T11:36:32.679 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:32 smithi179 ceph-mon[37907]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T11:36:32.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:32 smithi179 ceph-mon[29608]: osdmap e376: 8 total, 8 up, 8 in 2024-03-24T11:36:32.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:32 smithi179 ceph-mon[29608]: osdmap e377: 8 total, 8 up, 8 in 2024-03-24T11:36:32.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:32.784 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:32.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:32.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:32.944 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:32.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:33.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:33.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:33.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:33.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:33.222 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:33.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:33.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:33.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:33.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:33.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:33.464 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:36:33.464 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:36:33.464 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T11:36:33.464 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.609365 2024-03-24T11:36:33.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 0.6093647781364053 2024-03-24T11:36:33.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:33.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:33.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:33.512 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:33 smithi179 ceph-mon[37907]: pgmap v807: 178 pgs: 7 remapped+peering, 2 unknown, 169 active+clean; 332 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 496 KiB/s wr, 388 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:36:33.512 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:33 smithi179 ceph-mon[29608]: pgmap v807: 178 pgs: 7 remapped+peering, 2 unknown, 169 active+clean; 332 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 496 KiB/s wr, 388 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:36:33.595 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:33.645 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:33.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:33.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:33 smithi181 ceph-mon[33998]: pgmap v807: 178 pgs: 7 remapped+peering, 2 unknown, 169 active+clean; 332 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 496 KiB/s wr, 388 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T11:36:33.706 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:36:33.706 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:36:33.706 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:36:33.706 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:36:33.706 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:36:33.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:33.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:33.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:33.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:33.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:33.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:33.915 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:36:33.956 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:36:33.995 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:36:34.013 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:34.037 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:36:34.054 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:34.079 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:36:34.103 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:36:34.123 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:36:34.155 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:36:34.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:36:34.161 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:36:34.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:34.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:34.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:34.378 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:36:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:36:34.427 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.463 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:34.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:34.499 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Removed 110493 objects 2024-03-24T11:36:34.499 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Clean up completed and total clean up time :120.854 2024-03-24T11:36:34.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.589 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:34.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:34 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:34 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/555879231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:34.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:34 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:34 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/555879231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:34.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:34 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:34.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:34 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/555879231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:34.715 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:34.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:34.817 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:34.840 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:34.942 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:34.966 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:34.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:35.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:35.092 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:35.181 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:36:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:36:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:36:35.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:35.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:36:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.219 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:35.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:35.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:35.343 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:35.428 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:35.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:35 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:35 smithi179 ceph-mon[29608]: pgmap v808: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 96 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1007 B/s rd, 0 B/s wr, 1.05k op/s; 461/5196 objects misplaced (8.872%); 4.5 MiB/s, 72 objects/s recovering 2024-03-24T11:36:35.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:35 smithi179 ceph-mon[29608]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:36:35.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:35 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:35.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:35 smithi179 ceph-mon[37907]: pgmap v808: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 96 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1007 B/s rd, 0 B/s wr, 1.05k op/s; 461/5196 objects misplaced (8.872%); 4.5 MiB/s, 72 objects/s recovering 2024-03-24T11:36:35.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:35 smithi179 ceph-mon[37907]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:36:35.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:35.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:35.604 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:35.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:35 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:35 smithi181 ceph-mon[33998]: pgmap v808: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 96 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1007 B/s rd, 0 B/s wr, 1.05k op/s; 461/5196 objects misplaced (8.872%); 4.5 MiB/s, 72 objects/s recovering 2024-03-24T11:36:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:35 smithi181 ceph-mon[33998]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:36:35.706 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:35.729 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:35.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:35.855 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:35.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:35.985 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:35.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:35.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:36.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:36.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:36.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:36.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:36.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:36.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:36.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:36.370 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:36.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:36.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:36.500 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:36.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:36.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:36.631 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:36.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:36.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:36.761 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:36.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:36.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:36.878 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-24T11:36:36.879 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-24T11:36:36.889 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:36.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:36.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:37.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:37.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:37.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:37.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:37.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:37.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:37.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:37.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:37.511 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:37.532 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:37.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:37.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:37 smithi179 ceph-mon[29608]: pgmap v809: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 86 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 844 B/s rd, 0 B/s wr, 891 op/s; 461/4707 objects misplaced (9.794%); 3.8 MiB/s, 61 objects/s recovering 2024-03-24T11:36:37.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:37 smithi179 ceph-mon[37907]: pgmap v809: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 86 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 844 B/s rd, 0 B/s wr, 891 op/s; 461/4707 objects misplaced (9.794%); 3.8 MiB/s, 61 objects/s recovering 2024-03-24T11:36:37.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:37.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:37.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:37 smithi181 ceph-mon[33998]: pgmap v809: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 86 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 844 B/s rd, 0 B/s wr, 891 op/s; 461/4707 objects misplaced (9.794%); 3.8 MiB/s, 61 objects/s recovering 2024-03-24T11:36:37.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:36:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:37.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:37.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:38.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:38.043 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:38.138 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:38.145 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:38.176 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:38.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:38.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:38.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:38.436 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:38.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:38.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:38.668 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.693 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:38.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:38.794 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:38.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:38.924 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:38.951 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:38.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:39.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.076 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:39.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:39.177 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:39.205 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:39.308 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.333 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:39.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:39.434 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:39.458 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:39.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:39.584 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:39.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:39.693 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:39 smithi179 ceph-mon[29608]: pgmap v810: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 49 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 754 op/s; 323/2895 objects misplaced (11.157%); 2.9 MiB/s, 46 objects/s recovering 2024-03-24T11:36:39.693 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:39 smithi179 ceph-mon[37907]: pgmap v810: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 49 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 754 op/s; 323/2895 objects misplaced (11.157%); 2.9 MiB/s, 46 objects/s recovering 2024-03-24T11:36:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.712 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:39.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:39.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.838 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:39.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:39 smithi181 ceph-mon[33998]: pgmap v810: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 49 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 754 op/s; 323/2895 objects misplaced (11.157%); 2.9 MiB/s, 46 objects/s recovering 2024-03-24T11:36:39.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:39.967 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:39.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:40.069 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:40.093 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:40.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.223 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:40.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:40.324 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.348 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:40.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:40.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:40.477 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:40.579 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:40.602 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:40.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.730 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:40.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:40.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:40.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:40.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:40.988 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:40.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:41.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:41.117 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:41.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:41.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:41.243 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:41.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:41.345 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:41.368 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:41.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:41.470 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:41.473 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.2 primary-affinity to 0.609365 (8399352) 2024-03-24T11:36:41.498 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:41.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:41.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:41.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:41.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:41.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:41 smithi179 ceph-mon[29608]: pgmap v811: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 4.4 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 749 op/s; 61/777 objects misplaced (7.851%); 2.5 MiB/s, 41 objects/s recovering 2024-03-24T11:36:41.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4247440096' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]: dispatch 2024-03-24T11:36:41.690 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:41 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]: dispatch 2024-03-24T11:36:41.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:41 smithi179 ceph-mon[37907]: pgmap v811: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 4.4 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 749 op/s; 61/777 objects misplaced (7.851%); 2.5 MiB/s, 41 objects/s recovering 2024-03-24T11:36:41.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4247440096' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]: dispatch 2024-03-24T11:36:41.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:41 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]: dispatch 2024-03-24T11:36:41.691 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:36:41.691 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:36:41.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:41.754 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:41.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:41.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:41.881 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:41.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:41 smithi181 ceph-mon[33998]: pgmap v811: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 4.4 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 749 op/s; 61/777 objects misplaced (7.851%); 2.5 MiB/s, 41 objects/s recovering 2024-03-24T11:36:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4247440096' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]: dispatch 2024-03-24T11:36:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:41 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]: dispatch 2024-03-24T11:36:41.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:42.010 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:42.112 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:42.136 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:42.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:42.261 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:42.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:42.388 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:42.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:36:42.490 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:42.514 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:42.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:42.640 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:42.698 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:42 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]': finished 2024-03-24T11:36:42.699 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:42 smithi179 ceph-mon[29608]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T11:36:42.699 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:42 smithi179 ceph-mon[29608]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T11:36:42.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:42 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]': finished 2024-03-24T11:36:42.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:42 smithi179 ceph-mon[37907]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T11:36:42.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:42 smithi179 ceph-mon[37907]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T11:36:42.704 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:42.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:42.766 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:36:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:42.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:42.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:42.893 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:42.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:42 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6093647781364053}]': finished 2024-03-24T11:36:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:42 smithi181 ceph-mon[33998]: osdmap e378: 8 total, 8 up, 8 in 2024-03-24T11:36:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:42 smithi181 ceph-mon[33998]: osdmap e379: 8 total, 8 up, 8 in 2024-03-24T11:36:42.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:43.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:43.120 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:43.143 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:43.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:43.269 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:43.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.396 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:43.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:43.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:43.521 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:43.616 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:43 smithi179 ceph-mon[29608]: pgmap v813: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 4.4 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 677 op/s; 61/777 objects misplaced (7.851%); 2.3 MiB/s, 37 objects/s recovering 2024-03-24T11:36:43.617 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:43 smithi179 ceph-mon[37907]: pgmap v813: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 4.4 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 677 op/s; 61/777 objects misplaced (7.851%); 2.3 MiB/s, 37 objects/s recovering 2024-03-24T11:36:43.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.647 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:43.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:43.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:43 smithi181 ceph-mon[33998]: pgmap v813: 178 pgs: 6 active+remapped+backfill_toofull, 172 active+clean; 4.4 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 677 op/s; 61/777 objects misplaced (7.851%); 2.3 MiB/s, 37 objects/s recovering 2024-03-24T11:36:43.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:43.773 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:43.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:43.900 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:43.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:44.002 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:44.025 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:44.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:44.151 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:44.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:44.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:44.307 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_3' removed 2024-03-24T11:36:44.315 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:36:44.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:44.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:36:44.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:44.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.543 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:44.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:44.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:44.687 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:44.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:44.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/705227181' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:36:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2839885478' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:36:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:36:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/705227181' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:36:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:36:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:44 smithi181 ceph-mon[33998]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T11:36:44.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:44.937 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:44.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/705227181' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:36:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2839885478' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:36:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:36:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/705227181' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:36:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:36:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[29608]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T11:36:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/705227181' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:36:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2839885478' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:36:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:36:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:36:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/705227181' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:36:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:36:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:44 smithi179 ceph-mon[37907]: osdmap e380: 8 total, 8 up, 8 in 2024-03-24T11:36:45.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:45.062 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:45.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:45.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:45.267 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:36:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:36:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:36:45.289 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:45.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:45.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:45.438 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:45.540 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:45.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:45.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:45 smithi179 ceph-mon[37907]: pgmap v816: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:36:45.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:45 smithi179 ceph-mon[37907]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull) 2024-03-24T11:36:45.560 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:45.560 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:45 smithi179 ceph-mon[29608]: pgmap v816: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:36:45.560 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:45 smithi179 ceph-mon[29608]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull) 2024-03-24T11:36:45.564 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:45.666 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:45.690 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:45.725 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:45.725 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:45 smithi181 ceph-mon[33998]: pgmap v816: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:36:45.725 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:45 smithi181 ceph-mon[33998]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull) 2024-03-24T11:36:45.726 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:36:45.726 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:36:45.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:45.815 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:45.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:45.939 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:45.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:46.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.067 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:46.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:46.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:46.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:46.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:46.317 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:46.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:46.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:46.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:36:46.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:46.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:36:46.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:46 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:36:46.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:46.726 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T11:36:46.726 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T11:36:46.727 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-24T11:36:46.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-24T11:36:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:46.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:46.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:46.840 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:36:46.854 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:46 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:36:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:46 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:36:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:46 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:36:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:46 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:36:46.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:46.979 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:46.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:47.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:47.105 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:47.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.232 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:47.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:47.322 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:36:47.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:47.339 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:36:47.358 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:47.364 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:47.365 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:36:47.385 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:36:47.406 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:36:47.426 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:36:47.445 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:36:47.461 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:47.465 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:36:47.466 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:36:47.466 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:36:47.466 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:36:47.466 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:36:47.467 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:36:47.467 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:36:47.467 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:36:47.467 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:36:47.485 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:47.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:47 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:36:47.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:47 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:36:47.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:47 smithi181 ceph-mon[33998]: pgmap v817: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:36:47.559 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/158354355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:47.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:47.606 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:36:47.606 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:36:47.606 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[29608]: pgmap v817: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:36:47.606 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/158354355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:47.607 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:36:47.607 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:36:47.607 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[37907]: pgmap v817: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:36:47.607 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/158354355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:47.613 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:47.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:47.739 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:47.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.866 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:47.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:47.889 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:36:47.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:47.991 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:47.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:48.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:48.117 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:48.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:48.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:48.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:48.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:48.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:48.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:48.368 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:48.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:48.470 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:48.494 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:48.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:48.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:48.620 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:48.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:48 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:36:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:48 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:36:48.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:48 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:36:48.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:48 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:36:48.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:48.746 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:48.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:48.848 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:48.860 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:48 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:36:48.861 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:48 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:36:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:48.872 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:48.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:48.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:48.992 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:36:48.992 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:36:48.992 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:36:48.992 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:36:48.992 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:36:48.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:49.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.125 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:49.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:49.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:49.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:49.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:49.272 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:49.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:49.376 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:49.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.502 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:49.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:49.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:49.627 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:49.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:49.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:49.789 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:36:49.789 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:36:49.789 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: pgmap v818: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:36:49.790 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:49 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:36:49.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: pgmap v818: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:36:49.827 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: pgmap v818: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: osdmap e381: 8 total, 8 up, 8 in 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:36:49.828 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:49 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:36:49.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:49.881 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:49.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:49.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:50.004 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:50.016 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:36:50.016 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:36:50.016 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:36:50.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd out 4 2024-03-24T11:36:50.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:50.107 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:36:50.130 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:50.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:50.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:50.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:50.382 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:50.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:50.508 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:50.523 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_4' created 2024-03-24T11:36:50.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:50.634 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:36:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:36:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:36:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:36:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:36:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:36:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3047288624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:36:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:36:50.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3047288624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T11:36:50.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:50.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:50.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:50.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:50.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:36:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3047288624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T11:36:50.986 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:50.995 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:51.014 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:51.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:51.139 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:51.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:51.266 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:36:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.268 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:51.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:51.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:51.392 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:51.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:51.517 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:51.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-24T11:36:51.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:51.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:51.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:51.770 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: pgmap v820: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3047288624' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:36:51.830 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:51 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:36:51.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: pgmap v820: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3047288624' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:36:51.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: pgmap v820: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3047288624' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: osdmap e382: 8 total, 8 up, 8 in 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:36:51.885 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:51 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:36:51.894 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:51.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:51.944 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:51.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:52.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:52.093 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:36:52.093 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:36:52.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213990 10.c 10:33f1c950:::benchmark_data_smithi179_155357_object103494:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.291870+0000", 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.79923509, 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.065501571, 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213990 2024-03-24T11:36:52.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291870+0000", 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291870+0000", 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291872+0000", 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7579999999999999e-06 2024-03-24T11:36:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291881+0000", 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3889999999999994e-06 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291894+0000", 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2904e-05 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291904+0000", 2024-03-24T11:36:52.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7440000000000002e-06 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.293526+0000", 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001621707 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.293531+0000", 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6710000000000004e-06 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.503629+0000", 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.210097325 2024-03-24T11:36:52.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.503643+0000", 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3916e-05 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.357287+0000", 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.85364431699999999 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.357372+0000", 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4839999999999994e-05 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213992 10.1c 10:3e0f01f5:::benchmark_data_smithi179_155357_object103496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.298999+0000", 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.792106635, 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1072013119999999, 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.099 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213992 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.298999+0000", 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.298999+0000", 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299001+0000", 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3089999999999998e-06 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299008+0000", 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5530000000000004e-06 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299022+0000", 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3888e-05 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299032+0000", 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3309999999999993e-06 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504217+0000", 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20518531300000001 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504224+0000", 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0049999999999998e-06 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.382915+0000", 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.87869146499999995 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.383056+0000", 2024-03-24T11:36:52.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014087600000000001 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.383291+0000", 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000234916 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387031+0000", 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037401560000000001 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387043+0000", 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1688000000000001e-05 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.406147+0000", 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019103983000000001 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.406177+0000", 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0324999999999998e-05 2024-03-24T11:36:52.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.406200+0000", 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2504e-05 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213999 10.1c 10:3df129a2:::benchmark_data_smithi179_155357_object103503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.341073+0000", 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.750031801999999, 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.067161144, 2024-03-24T11:36:52.106 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213999 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341073+0000", 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341073+0000", 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341076+0000", 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5610000000000001e-06 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341086+0000", 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9930000000000007e-06 2024-03-24T11:36:52.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341116+0000", 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0023999999999999e-05 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341132+0000", 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6079999999999999e-05 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504245+0000", 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.163112966 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504251+0000", 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5860000000000004e-06 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.385818+0000", 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.88156710900000002 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.385938+0000", 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120684 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.386116+0000", 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017782799999999999 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.389597+0000", 2024-03-24T11:36:52.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034809900000000002 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.389605+0000", 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2690000000000002e-06 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408134+0000", 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018529747999999999 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408190+0000", 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6209999999999999e-05 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408235+0000", 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4095999999999999e-05 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214003 10.21 10:84cdce02:::benchmark_data_smithi179_155357_object103507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.544295+0000", 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.546810356000002, 2024-03-24T11:36:52.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82516384700000001, 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214003 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544295+0000", 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544295+0000", 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544297+0000", 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8789999999999999e-06 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544304+0000", 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3440000000000003e-06 2024-03-24T11:36:52.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544326+0000", 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1441999999999999e-05 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544338+0000", 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2578999999999999e-05 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544423+0000", 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4643999999999998e-05 2024-03-24T11:36:52.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544436+0000", 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2926e-05 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360241+0000", 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81580506799999997 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360368+0000", 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000127272 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360615+0000", 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024656500000000002 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.363504+0000", 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002889015 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.363650+0000", 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014628300000000001 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369348+0000", 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005698196 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369410+0000", 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1618000000000005e-05 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369459+0000", 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9016e-05 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214004 10.1c 10:3bbddca2:::benchmark_data_smithi179_155357_object103508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.553160+0000", 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.537945539999999, 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.85520687100000004, 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.120 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.121 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.121 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.121 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.121 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214004 2024-03-24T11:36:52.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:52.122 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553160+0000", 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553160+0000", 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553161+0000", 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3829999999999999e-06 2024-03-24T11:36:52.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553166+0000", 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1560000000000003e-06 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553179+0000", 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.325e-05 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553188+0000", 2024-03-24T11:36:52.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7269999999999996e-06 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553256+0000", 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8279000000000001e-05 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553267+0000", 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0647e-05 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387067+0000", 2024-03-24T11:36:52.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83380036400000002 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387194+0000", 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126978 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387331+0000", 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013626799999999999 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.389607+0000", 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022766819999999999 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.390696+0000", 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010887539999999999 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408280+0000", 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017583863000000002 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408329+0000", 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9282e-05 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408367+0000", 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7237999999999998e-05 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215487 10.23 10.e5a750a3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.360194+0000", 2024-03-24T11:36:52.128 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.730911030000001, 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96463855099999996, 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215487 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360194+0000", 2024-03-24T11:36:52.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360194+0000", 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360195+0000", 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2139999999999999e-06 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360201+0000", 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4500000000000003e-06 2024-03-24T11:36:52.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360211+0000", 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0478e-05 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360217+0000", 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9109999999999998e-06 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360284+0000", 2024-03-24T11:36:52.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6818999999999998e-05 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360294+0000", 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0278000000000001e-05 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.324822+0000", 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96452802900000001 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.324833+0000", 2024-03-24T11:36:52.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0372e-05 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215490 10.21 10:84f9e374:::benchmark_data_smithi179_155357_object104994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.360982+0000", 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.730123177999999, 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.070579167, 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.133 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215490 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360982+0000", 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360982+0000", 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360983+0000", 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5400000000000001e-07 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360986+0000", 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.613e-06 2024-03-24T11:36:52.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360996+0000", 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5400000000000001e-06 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.361002+0000", 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.66e-06 2024-03-24T11:36:52.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.361062+0000", 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0346999999999997e-05 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.361075+0000", 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2673e-05 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.320218+0000", 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95914328800000004 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.320234+0000", 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6316000000000001e-05 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.419190+0000", 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.098956238 2024-03-24T11:36:52.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.419341+0000", 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015083699999999999 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2,7", 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.419576+0000", 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023469699999999999 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.420780+0000", 2024-03-24T11:36:52.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001203976 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.422186+0000", 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014063859999999999 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.425236+0000", 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030498109999999999 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.431359+0000", 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061229530000000004 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.431439+0000", 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9555000000000007e-05 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.431561+0000", 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012262300000000001 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.141 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215525 10.1d 10:bd921279:::benchmark_data_smithi179_155357_object105029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.407932+0000", 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.683172929000001, 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0504010400000001, 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.142 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215525 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407932+0000", 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407932+0000", 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407933+0000", 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0860000000000001e-06 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407938+0000", 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.25e-06 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407948+0000", 2024-03-24T11:36:52.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0584999999999999e-05 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407957+0000", 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3070000000000003e-06 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.408014+0000", 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7741000000000002e-05 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.408022+0000", 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8089999999999993e-06 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323440+0000", 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91541776699999999 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323452+0000", 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1759999999999999e-05 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.450011+0000", 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1265593039999999 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.450121+0000", 2024-03-24T11:36:52.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109869 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6", 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.450362+0000", 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024074900000000001 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451902+0000", 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015406549999999999 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458058+0000", 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061556670000000001 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458108+0000", 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0351e-05 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458161+0000", 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3297000000000001e-05 2024-03-24T11:36:52.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458224+0000", 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2209e-05 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458333+0000", 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109634 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215434 10.1c 10:3e049106:::benchmark_data_smithi179_155357_object104938:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.322005+0000", 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.769100011999999, 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.119395546, 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215434 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322005+0000", 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322005+0000", 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322007+0000", 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9879999999999999e-06 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322012+0000", 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6700000000000002e-06 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322925+0000", 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091357500000000002 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322939+0000", 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4015e-05 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325695+0000", 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002755118 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325704+0000", 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7499999999999998e-06 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441300+0000", 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.115596145 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441401+0000", 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010028500000000001 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215438 10.1c 10:3d6c033a:::benchmark_data_smithi179_155357_object104942:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325611+0000", 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.765494160999999, 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.13719357, 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215438 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325611+0000", 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325611+0000", 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325612+0000", 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2160000000000001e-06 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325618+0000", 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6280000000000002e-06 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325632+0000", 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4453999999999999e-05 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325643+0000", 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0468999999999999e-05 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325784+0000", 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014096700000000001 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325792+0000", 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8520000000000004e-06 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441694+0000", 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.115902816 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441826+0000", 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013180399999999999 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.462624+0000", 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020798179999999999 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.462805+0000", 2024-03-24T11:36:52.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018018400000000001 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215480 10.1c 10:3cfdbf27:::benchmark_data_smithi179_155357_object104984:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325659+0000", 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.765446327999999, 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.125743108, 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.161 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215480 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325659+0000", 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325659+0000", 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325660+0000", 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3e-07 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325663+0000", 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4410000000000002e-06 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325671+0000", 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2800000000000003e-06 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325678+0000", 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.041e-06 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325818+0000", 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013960200000000001 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325825+0000", 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2760000000000003e-06 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441915+0000", 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1160902100000001 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.442023+0000", 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010732 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6,7", 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.442240+0000", 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000217004 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447810+0000", 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055700369999999999 2024-03-24T11:36:52.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447934+0000", 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012472699999999999 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447971+0000", 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6705999999999997e-05 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.448006+0000", 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5228999999999998e-05 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451173+0000", 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003166678 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451270+0000", 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6434999999999994e-05 2024-03-24T11:36:52.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451402+0000", 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000132492 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215504 10.1c 10:3d911fcf:::benchmark_data_smithi179_155357_object105008:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.765417436, 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.129241637, 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.169 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215504 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7199999999999999e-07 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325691+0000", 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9189999999999999e-06 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325699+0000", 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4359999999999996e-06 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325706+0000", 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9110000000000004e-06 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325853+0000", 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014751599999999999 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325861+0000", 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7319999999999998e-06 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443267+0000", 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1174060210000001 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443382+0000", 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115062 2024-03-24T11:36:52.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6,7", 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443550+0000", 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016815499999999999 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447853+0000", 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043024960000000003 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.448043+0000", 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000190173 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451443+0000", 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003400368 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454549+0000", 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003105642 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454761+0000", 2024-03-24T11:36:52.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021172199999999999 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454813+0000", 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.253e-05 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454929+0000", 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116382 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.176 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215517 10.1c 10:39423bd9:::benchmark_data_smithi179_155357_object105021:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325817+0000", 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.765287843999999, 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1324648829999999, 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215517 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325817+0000", 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325817+0000", 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325818+0000", 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4600000000000004e-07 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325823+0000", 2024-03-24T11:36:52.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3869999999999997e-06 2024-03-24T11:36:52.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325834+0000", 2024-03-24T11:36:52.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1126e-05 2024-03-24T11:36:52.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325842+0000", 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.507e-06 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325887+0000", 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4697000000000003e-05 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325894+0000", 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8600000000000004e-06 2024-03-24T11:36:52.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446112+0000", 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.120217829 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446229+0000", 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117335 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6,7", 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446435+0000", 2024-03-24T11:36:52.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000205859 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451868+0000", 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054331620000000001 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454562+0000", 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026939120000000001 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454719+0000", 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015704400000000001 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454965+0000", 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000245937 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.455045+0000", 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0037000000000006e-05 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458134+0000", 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030891690000000001 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458282+0000", 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000148276 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.185 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217810 10.1d 10:b9e8646c:::benchmark_data_smithi179_155357_object107314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.390708+0000", 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.7003971, 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98764608099999995, 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217810 2024-03-24T11:36:52.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390708+0000", 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390708+0000", 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390709+0000", 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8899999999999999e-07 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390713+0000", 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3579999999999996e-06 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390732+0000", 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8569000000000001e-05 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390738+0000", 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8710000000000002e-06 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390789+0000", 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1566000000000001e-05 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390798+0000", 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0119999999999994e-06 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.374933+0000", 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98413457699999995 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.375047+0000", 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011410599999999999 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.375249+0000", 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000202258 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378145+0000", 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028957010000000001 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378196+0000", 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1397000000000003e-05 2024-03-24T11:36:52.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378260+0000", 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3486999999999997e-05 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378313+0000", 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3588999999999998e-05 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378354+0000", 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0901e-05 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217862 10.1d 10:ba4afeec:::benchmark_data_smithi179_155357_object107366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.430683+0000", 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.660422630999999, 2024-03-24T11:36:52.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.94900074700000003, 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217862 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430683+0000", 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430683+0000", 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430684+0000", 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1060000000000001e-06 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430688+0000", 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6809999999999997e-06 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430699+0000", 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0302000000000001e-05 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430705+0000", 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0440000000000001e-06 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430763+0000", 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8712000000000003e-05 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430774+0000", 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0396e-05 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.376471+0000", 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.94569716100000001 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.376549+0000", 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7528999999999996e-05 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.376708+0000", 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000159922 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378426+0000", 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017170810000000001 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379509+0000", 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010837690000000001 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379583+0000", 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3565999999999993e-05 2024-03-24T11:36:52.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379640+0000", 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6780000000000002e-05 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379683+0000", 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3698000000000001e-05 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217905 10.1d 10:bd552d80:::benchmark_data_smithi179_155357_object107409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.631873708000001, 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.92063982300000002, 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217905 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459234+0000", 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9010000000000001e-06 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459238+0000", 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5589999999999999e-06 2024-03-24T11:36:52.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459242+0000", 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3639999999999999e-06 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459287+0000", 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4901000000000003e-05 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459295+0000", 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8759999999999994e-06 2024-03-24T11:36:52.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.377420+0000", 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91812414499999995 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.377483+0000", 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3854000000000002e-05 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.377593+0000", 2024-03-24T11:36:52.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109219 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379524+0000", 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019309500000000001 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379739+0000", 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021520099999999999 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379780+0000", 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0915999999999999e-05 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379831+0000", 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0844000000000003e-05 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379871+0000", 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0757000000000003e-05 2024-03-24T11:36:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217927 10.1d 10:bcda50cf:::benchmark_data_smithi179_155357_object107431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.476986+0000", 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.614119162000001, 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90311201900000004, 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.208 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217927 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476986+0000", 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476986+0000", 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476988+0000", 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.502e-06 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476994+0000", 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6069999999999996e-06 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477016+0000", 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1611e-05 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477024+0000", 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6910000000000003e-06 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477077+0000", 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2475999999999997e-05 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477083+0000", 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.297e-06 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378453+0000", 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90136997900000004 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378556+0000", 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103014 2024-03-24T11:36:52.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378686+0000", 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012929 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379933+0000", 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001247791 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379984+0000", 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1115000000000001e-05 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380003+0000", 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8706999999999999e-05 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380053+0000", 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9873000000000002e-05 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380098+0000", 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5065999999999998e-05 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217957 10.1d 10:ba42563d:::benchmark_data_smithi179_155357_object107461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.497518+0000", 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.593587028999998, 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.89107709099999999, 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.215 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217957 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497518+0000", 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497518+0000", 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497519+0000", 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6300000000000005e-07 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497523+0000", 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7960000000000002e-06 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497532+0000", 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.781e-06 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497540+0000", 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.323e-06 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497628+0000", 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8265999999999998e-05 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497638+0000", 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0192e-05 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380134+0000", 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.88249543799999997 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380244+0000", 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011079 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380416+0000", 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171853 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.383884+0000", 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003468151 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.387184+0000", 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032990710000000002 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388495+0000", 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001311638 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388555+0000", 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0272999999999997e-05 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388595+0000", 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9855999999999999e-05 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217989 10.1d 10:bd18519e:::benchmark_data_smithi179_155357_object107493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.521620+0000", 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.569485050000001, 2024-03-24T11:36:52.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.867147318, 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217989 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521620+0000", 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521620+0000", 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521621+0000", 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7899999999999998e-07 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521624+0000", 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3639999999999999e-06 2024-03-24T11:36:52.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521633+0000", 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3510000000000002e-06 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521639+0000", 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.545e-06 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521698+0000", 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8452999999999998e-05 2024-03-24T11:36:52.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521707+0000", 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5540000000000003e-06 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.381365+0000", 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.85965779099999995 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.381469+0000", 2024-03-24T11:36:52.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104182 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.381640+0000", 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000170636 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.383890+0000", 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022498219999999998 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.387314+0000", 2024-03-24T11:36:52.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034244560000000002 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388662+0000", 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001348094 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388718+0000", 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5823999999999999e-05 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388767+0000", 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9567000000000002e-05 2024-03-24T11:36:52.227 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:218013 10.1d 10:be135f22:::benchmark_data_smithi179_155357_object107517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.547096+0000", 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.544009171999999, 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.84178786999999999, 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 218013 2024-03-24T11:36:52.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547096+0000", 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547096+0000", 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547097+0000", 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0499999999999999e-06 2024-03-24T11:36:52.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547102+0000", 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4460000000000003e-06 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547114+0000", 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2875e-05 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547122+0000", 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7859999999999997e-06 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547172+0000", 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9935999999999998e-05 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547178+0000", 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5600000000000001e-06 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.382589+0000", 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83541153800000001 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.382696+0000", 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106716 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.382884+0000", 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000187596 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.385085+0000", 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022019309999999999 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.387360+0000", 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002274781 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388810+0000", 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014493189999999999 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388851+0000", 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1108e-05 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:36:52.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388884+0000", 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3228000000000001e-05 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:36:52.234 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:36:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:52.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:52.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:52.360 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:52.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.488 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:52.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:52.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:52.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:52.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.743 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:52.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:52.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:52.868 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:52 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:36:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:52 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:36:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:52 smithi181 ceph-mon[33998]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T11:36:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:52 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:36:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:52 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:36:52.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:36:52.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:36:52.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[37907]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T11:36:52.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:36:52.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:36:52.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:36:52.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:36:52.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[29608]: osdmap e383: 8 total, 8 up, 8 in 2024-03-24T11:36:52.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:36:52.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:52 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:36:52.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:52.994 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:52.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:53.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:53.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:53.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:53.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:53.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:53.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:53.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:53.371 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:53.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:53.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:36:53.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:53.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:53.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:53.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:53.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:53.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:53.698 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:36:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:36:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:36:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[29608]: pgmap v823: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[29608]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:36:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[29608]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T11:36:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:36:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:36:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:36:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:36:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[37907]: pgmap v823: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[37907]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:36:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[37907]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T11:36:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:36:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:53 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:36:53.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:53.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:53.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:53.820 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:53 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:36:53.820 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:53 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:36:53.820 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:53 smithi181 ceph-mon[33998]: pgmap v823: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:36:53.820 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:53 smithi181 ceph-mon[33998]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:36:53.820 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:53 smithi181 ceph-mon[33998]: osdmap e384: 8 total, 8 up, 8 in 2024-03-24T11:36:53.820 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:53 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:36:53.820 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:53 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:36:53.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:53.896 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:53.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:53.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:54.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:54.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:54.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:54.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:54.274 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:54.376 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:54.400 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:54.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:36:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:36:54.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:54.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:54.628 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:54.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[29608]: 6.1b scrub starts 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[29608]: 6.1b scrub ok 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[29608]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[37907]: 6.1b scrub starts 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[37907]: 6.1b scrub ok 2024-03-24T11:36:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:36:54.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:36:54.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:54 smithi179 ceph-mon[37907]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T11:36:54.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.780 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:54.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:54.841 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:54 smithi181 ceph-mon[33998]: 6.1b scrub starts 2024-03-24T11:36:54.940 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:54 smithi181 ceph-mon[33998]: 6.1b scrub ok 2024-03-24T11:36:54.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:54 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:36:54.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:54 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:36:54.941 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:54 smithi181 ceph-mon[33998]: osdmap e385: 8 total, 8 up, 8 in 2024-03-24T11:36:54.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:54.965 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:54.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:55.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:55.071 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:36:55.091 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:55.181 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:36:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:36:55.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:55.217 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:55.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:55.343 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:55.362 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:36:55.362 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:36:55.362 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-03-24T11:36:55.362 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:36:55.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:55.470 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:55.549 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:36:55.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.596 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:55.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:55.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:36:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:36:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:36:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: pgmap v826: 158 pgs: 5 unknown, 153 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3724959495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: pgmap v826: 158 pgs: 5 unknown, 153 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:36:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3724959495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:55.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:55 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:55.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:36:55.822 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.847 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:55.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: pgmap v826: 158 pgs: 5 unknown, 153 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3724959495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:55 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:36:55.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:55.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:55.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:56.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:56.096 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:56.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:36:56.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:56.222 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:56.284 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:36:56.303 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:36:56.322 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:36:56.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:56.342 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:36:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:56.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:56.362 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:36:56.382 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:36:56.403 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:36:56.418 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:36:56.423 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:36:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:36:56.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:56.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:56.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:56.599 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:56.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:36:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:36:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T11:36:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:36:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:36:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:36:56.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: 3.12 deep-scrub starts 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: 3.12 deep-scrub ok 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/177847064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: 3.12 deep-scrub starts 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: 3.12 deep-scrub ok 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:36:56.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:56 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/177847064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:56.724 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:56.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:36:56.849 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: osdmap e386: 8 total, 8 up, 8 in 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: 3.12 deep-scrub starts 2024-03-24T11:36:56.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: 3.12 deep-scrub ok 2024-03-24T11:36:56.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:36:56.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:56 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/177847064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:36:56.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:56.977 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:56.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:57.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:36:57.101 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:36:57.203 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.228 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:57.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:57.329 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:57.352 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:57.454 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:57.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29604]: 2024-03-24T11:36:57.156+0000 7f36b2553700 -1 mon.a@0(leader).osd e386 definitely_dead 0 2024-03-24T11:36:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.478 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:57.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:57.565 INFO:teuthology.orchestra.run.smithi179.stderr:marked out osd.4. 2024-03-24T11:36:57.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:57.605 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:57.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:57.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:57.820 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: pgmap v828: 158 pgs: 158 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 5.11 deep-scrub starts 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 5.11 deep-scrub ok 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2989437581' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-24T11:36:57.821 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[37907]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: pgmap v828: 158 pgs: 158 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:36:57.822 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 5.11 deep-scrub starts 2024-03-24T11:36:57.823 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 5.11 deep-scrub ok 2024-03-24T11:36:57.823 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:36:57.823 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:36:57.823 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2989437581' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-24T11:36:57.823 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-24T11:36:57.823 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:57 smithi179 ceph-mon[29608]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-24T11:36:57.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:57.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: pgmap v828: 158 pgs: 158 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 5.11 deep-scrub starts 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 5.11 deep-scrub ok 2024-03-24T11:36:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:36:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:36:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2989437581' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-24T11:36:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-24T11:36:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:57 smithi181 ceph-mon[33998]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-24T11:36:57.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:57.984 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:57.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:58.085 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:58.109 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:58.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:58.211 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:58.234 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:58.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:58.244 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:36:58.244 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:36:58.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:36:58.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:58.361 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:58.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:36:58.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:58.486 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:58.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:58.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:58.630 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:58.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:58.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:58.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:58.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:58.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:36:58.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:58.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:36:58.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:58 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:36:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:58 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:36:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:58 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-24T11:36:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:58 smithi181 ceph-mon[33998]: osdmap e387: 8 total, 8 up, 7 in 2024-03-24T11:36:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:58 smithi181 ceph-mon[33998]: osdmap e388: 8 total, 8 up, 7 in 2024-03-24T11:36:58.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:36:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:36:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:36:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-24T11:36:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[29608]: osdmap e387: 8 total, 8 up, 7 in 2024-03-24T11:36:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[29608]: osdmap e388: 8 total, 8 up, 7 in 2024-03-24T11:36:58.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:36:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:36:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-24T11:36:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[37907]: osdmap e387: 8 total, 8 up, 7 in 2024-03-24T11:36:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:58 smithi179 ceph-mon[37907]: osdmap e388: 8 total, 8 up, 7 in 2024-03-24T11:36:58.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:59.006 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:59.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:36:59.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:36:59.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:59.243 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:36:59.258 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:59.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:59.383 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:59.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:59.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:59.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:36:59.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:36:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: pgmap v830: 158 pgs: 13 peering, 1 unknown, 144 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: osdmap e389: 8 total, 8 up, 7 in 2024-03-24T11:36:59.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:36:59 smithi181 ceph-mon[33998]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 83'128 2024-03-24T11:36:59.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:59.761 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:59.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:36:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:36:59.888 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:36:59.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: pgmap v830: 158 pgs: 13 peering, 1 unknown, 144 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: osdmap e389: 8 total, 8 up, 7 in 2024-03-24T11:36:59.905 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[29608]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 83'128 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: pgmap v830: 158 pgs: 13 peering, 1 unknown, 144 active+clean; 582 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: osdmap e389: 8 total, 8 up, 7 in 2024-03-24T11:36:59.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:36:59 smithi179 ceph-mon[37907]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 83'128 2024-03-24T11:36:59.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:36:59.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:00.039 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:00.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:00.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:00.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:00.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:00.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.418 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:00.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:00.476 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:00.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:00.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:00.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:00.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:00.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: 5.0 scrub starts 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: 5.0 scrub ok 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: 1.5 deep-scrub starts 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: 1.5 deep-scrub ok 2024-03-24T11:37:00.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:00 smithi181 ceph-mon[33998]: osdmap e390: 8 total, 8 up, 7 in 2024-03-24T11:37:00.668 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:00.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:00.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:37:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.794 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:00.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: 5.0 scrub starts 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: 5.0 scrub ok 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: 1.5 deep-scrub starts 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: 1.5 deep-scrub ok 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[29608]: osdmap e390: 8 total, 8 up, 7 in 2024-03-24T11:37:00.841 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: 5.0 scrub starts 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: 5.0 scrub ok 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: 1.5 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: 1.5 deep-scrub starts 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: 1.5 deep-scrub ok 2024-03-24T11:37:00.842 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:00 smithi179 ceph-mon[37907]: osdmap e390: 8 total, 8 up, 7 in 2024-03-24T11:37:00.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:00.920 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:00.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:01.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:01.046 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:01.070 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:37:01.070 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:37:01.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:01.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:01.273 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:01.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:01.307 INFO:teuthology.orchestra.run.smithi179.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-24T11:37:01.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:01.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:01.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:01.550 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:01.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:01.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:01.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 4.1d scrub starts 2024-03-24T11:37:01.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 4.1d scrub ok 2024-03-24T11:37:01.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:37:01.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:37:01.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: pgmap v834: 158 pgs: 4 activating+degraded, 16 activating, 1 remapped+peering, 21 peering, 1 unknown, 115 active+clean; 609 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 2.2 KiB/s rd, 27 KiB/s wr, 3 op/s; 33/594 objects degraded (5.556%); 20 KiB/s, 6 objects/s recovering 2024-03-24T11:37:01.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:37:01.698 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:37:01.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/88153123' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:01.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:37:01.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:37:01.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: Health check failed: Degraded data redundancy: 33/594 objects degraded (5.556%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:37:01.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/88153123' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:01.699 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:01 smithi181 ceph-mon[33998]: osdmap e391: 8 total, 8 up, 7 in 2024-03-24T11:37:01.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:01.802 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:01.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:37:01.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 4.1d scrub starts 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 4.1d scrub ok 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: pgmap v834: 158 pgs: 4 activating+degraded, 16 activating, 1 remapped+peering, 21 peering, 1 unknown, 115 active+clean; 609 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 2.2 KiB/s rd, 27 KiB/s wr, 3 op/s; 33/594 objects degraded (5.556%); 20 KiB/s, 6 objects/s recovering 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/88153123' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:37:01.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: Health check failed: Degraded data redundancy: 33/594 objects degraded (5.556%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/88153123' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[37907]: osdmap e391: 8 total, 8 up, 7 in 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 4.1d scrub starts 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 4.1d scrub ok 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: pgmap v834: 158 pgs: 4 activating+degraded, 16 activating, 1 remapped+peering, 21 peering, 1 unknown, 115 active+clean; 609 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 2.2 KiB/s rd, 27 KiB/s wr, 3 op/s; 33/594 objects degraded (5.556%); 20 KiB/s, 6 objects/s recovering 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/88153123' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:37:01.932 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: Health check failed: Degraded data redundancy: 33/594 objects degraded (5.556%), 4 pgs degraded (PG_DEGRADED) 2024-03-24T11:37:01.933 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/88153123' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:37:01.933 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:01 smithi179 ceph-mon[29608]: osdmap e391: 8 total, 8 up, 7 in 2024-03-24T11:37:01.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:01.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:01.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:02.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:02.068 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:02.160 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:02.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:02.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:02.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:02.321 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:02.386 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:02.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:02.447 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:02.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.573 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:02.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:02.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:02.698 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:02.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:02.823 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 3.8 scrub starts 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 3.8 scrub ok 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 4.19 scrub starts 2024-03-24T11:37:02.876 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[29608]: 4.19 scrub ok 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 3.8 scrub starts 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 3.8 scrub ok 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 4.19 scrub starts 2024-03-24T11:37:02.877 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:02 smithi179 ceph-mon[37907]: 4.19 scrub ok 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 3.8 scrub starts 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 3.8 scrub ok 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 4.19 scrub starts 2024-03-24T11:37:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:02 smithi181 ceph-mon[33998]: 4.19 scrub ok 2024-03-24T11:37:02.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:02.950 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:02.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:02.959 DEBUG:teuthology.orchestra.run.smithi179:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-24T11:37:02.995 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T11:37:02.995 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T11:37:03.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:03.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:03.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:03.131 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:hints = 1 2024-03-24T11:37:03.131 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T11:37:03.131 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object prefix: benchmark_data_smithi179_178435 2024-03-24T11:37:03.131 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:37:03.131 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 0 0 0 0 0 0 - 0 2024-03-24T11:37:03.178 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:03.215 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:03.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:03.317 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:03.356 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:03.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:03.457 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:03.500 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:03.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:03.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:03.648 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:03.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:37:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:37:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[29608]: pgmap v836: 158 pgs: 4 activating+degraded, 16 activating, 1 remapped+peering, 21 peering, 1 unknown, 115 active+clean; 609 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 2 op/s; 33/594 objects degraded (5.556%); 20 KiB/s, 6 objects/s recovering 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[29608]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[37907]: pgmap v836: 158 pgs: 4 activating+degraded, 16 activating, 1 remapped+peering, 21 peering, 1 unknown, 115 active+clean; 609 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 2 op/s; 33/594 objects degraded (5.556%); 20 KiB/s, 6 objects/s recovering 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[37907]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:37:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:03 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:37:03.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:03.784 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:03.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:03.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:03 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:37:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:03 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:37:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:03 smithi181 ceph-mon[33998]: pgmap v836: 158 pgs: 4 activating+degraded, 16 activating, 1 remapped+peering, 21 peering, 1 unknown, 115 active+clean; 609 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 2 op/s; 33/594 objects degraded (5.556%); 20 KiB/s, 6 objects/s recovering 2024-03-24T11:37:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:03 smithi181 ceph-mon[33998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:37:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:03 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:37:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:03 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:37:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:03.926 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:03.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:04.009 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:37:04.009 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:37:04.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:04.059 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:04.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:04.131 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 1 16 2952 2936 183.49 183.5 0.00510717 0.00541859 2024-03-24T11:37:04.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:04.212 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:04.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:04.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:04.357 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:04.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:04.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:37:04.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:04.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:04.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:04.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:04.624 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:04.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:04.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:04.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:37:04.740 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:37:04.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:04.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:04.850 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2975447243' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[29608]: Health check update: Reduced data availability: 14 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:04.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:04.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:37:04.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:37:04.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2975447243' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:37:04.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:37:04.852 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:04 smithi179 ceph-mon[37907]: Health check update: Reduced data availability: 14 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-24T11:37:04.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:04.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:04.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2975447243' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:37:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:37:04.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:04 smithi181 ceph-mon[33998]: Health check update: Reduced data availability: 14 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-24T11:37:04.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:37:04.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:37:04.933 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:37:04.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T11:37:04.934 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:37:04.977 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:37:04.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:05.022 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:37:05.042 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:05.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:05.063 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:37:05.086 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:05.102 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:37:05.103 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:37:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:37:05.132 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 2 16 5612 5596 174.83 166.25 0.00627358 0.00570349 2024-03-24T11:37:05.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:05.147 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:37:05.152 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:37:05.153 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:37:05.153 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:37:05.153 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:37:05.153 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:37:05.153 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:37:05.153 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:37:05.153 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:37:05.189 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:05.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:05.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:05.323 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:05.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:05.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:05.469 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:05.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:05.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:05.609 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:05.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:05.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:05.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:05.745 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:05.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:05.846 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: pgmap v837: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 41 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 6.9 MiB/s wr, 111 op/s; 433 B/s, 14 objects/s recovering 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 8 pgs peering) 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/594 objects degraded (5.556%), 4 pgs degraded) 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:37:05.877 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: osdmap e392: 8 total, 8 up, 7 in 2024-03-24T11:37:05.878 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:05 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4197828350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:05.886 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:05.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: pgmap v837: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 41 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 6.9 MiB/s wr, 111 op/s; 433 B/s, 14 objects/s recovering 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 8 pgs peering) 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/594 objects degraded (5.556%), 4 pgs degraded) 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: osdmap e392: 8 total, 8 up, 7 in 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4197828350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: pgmap v837: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 41 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 6.9 MiB/s wr, 111 op/s; 433 B/s, 14 objects/s recovering 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 8 pgs peering) 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/594 objects degraded (5.556%), 4 pgs degraded) 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: osdmap e392: 8 total, 8 up, 7 in 2024-03-24T11:37:05.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:05 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4197828350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:05.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:06.020 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:06.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:06.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:06.133 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 3 16 8468 8452 175.962 178.5 0.0104599 0.00567358 2024-03-24T11:37:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:06.165 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:06.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:06.266 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:06.304 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:06.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:06.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:06.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:06.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:06.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:06.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:06.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:06.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:06.749 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:06.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:06.764 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:06.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:06.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:06.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:06.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:07.014 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:07.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:07.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:07.134 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 4 16 11065 11049 172.531 162.312 0.00629635 0.00578667 2024-03-24T11:37:07.158 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:07.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:07.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:07.306 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:07.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:07.339 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:37:07.339 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:37:07.339 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:37:07.339 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:37:07.339 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:37:07.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:07.440 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:07.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:07.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:07.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:07.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:07.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:07.722 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:07.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:07.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:37:07.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:07.890 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:07.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:07.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:07 smithi181 ceph-mon[33998]: pgmap v839: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 145 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 24 MiB/s wr, 377 op/s; 0 B/s, 9 objects/s recovering 2024-03-24T11:37:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:07 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:07 smithi179 ceph-mon[29608]: pgmap v839: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 145 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 24 MiB/s wr, 377 op/s; 0 B/s, 9 objects/s recovering 2024-03-24T11:37:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:07 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:07 smithi179 ceph-mon[37907]: pgmap v839: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 145 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 24 MiB/s wr, 377 op/s; 0 B/s, 9 objects/s recovering 2024-03-24T11:37:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:07 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:37:07.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:08.023 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:08.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:08.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T11:37:08.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:08.138 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 5 16 13228 13212 164.923 135.188 0.00806013 0.00605218 2024-03-24T11:37:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:08.212 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:08.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:08.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:08.350 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:08.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:08.448 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:08.452 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:08.490 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:08.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:08.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:08.646 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:08.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:08.713 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:37:08.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:08.755 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:08.820 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:08.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:08.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:08 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:08.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:08 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:08.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:08 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:08.896 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:08 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:08.896 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:08 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:08.896 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:08 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:08 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:37:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:08 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:37:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:08 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:08.921 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:08.951 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:08.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:09.053 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:09.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:09.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:09.138 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 6 16 15214 15198 158.127 124.125 0.00433137 0.00631549 2024-03-24T11:37:09.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:09.243 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:09.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:09.314 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T11:37:09.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:09.374 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:09.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:09.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:09.519 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:09.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:09.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:09 smithi181 ceph-mon[33998]: pgmap v840: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 349 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s rd, 44 MiB/s wr, 696 op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:37:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:09 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:37:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:09 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:37:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3907511633' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:37:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:37:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:37:09.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:09 smithi181 ceph-mon[33998]: osdmap e393: 8 total, 8 up, 7 in 2024-03-24T11:37:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:09.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:09.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:09.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:09.791 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:09.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:09.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:09.932 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:09.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[29608]: pgmap v840: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 349 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s rd, 44 MiB/s wr, 696 op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:37:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:37:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:37:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3907511633' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:37:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:37:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:37:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[29608]: osdmap e393: 8 total, 8 up, 7 in 2024-03-24T11:37:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[37907]: pgmap v840: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 349 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s rd, 44 MiB/s wr, 696 op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:37:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:37:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:37:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3907511633' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:37:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:37:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:37:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:09 smithi179 ceph-mon[37907]: osdmap e393: 8 total, 8 up, 7 in 2024-03-24T11:37:10.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:10.062 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:10.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:10.138 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 7 16 17482 17466 155.785 141.75 0.00470356 0.00641255 2024-03-24T11:37:10.164 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:10.204 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:10.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:10.306 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:10.347 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:10.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:10.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:10.491 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:10.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:10.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:10.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:10.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:10.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213978 10.0 10:028c0284:::benchmark_data_smithi179_155357_object103482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.279702+0000", 2024-03-24T11:37:10.687 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.405002394999997, 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25125808900000002, 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213978 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.279702+0000", 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.279702+0000", 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.279704+0000", 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5409999999999999e-06 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.279708+0000", 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2599999999999999e-06 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.279718+0000", 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.004e-05 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.279734+0000", 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6152000000000001e-05 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.279793+0000", 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8445e-05 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280039+0000", 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000246588 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,4", 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280495+0000", 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00045521100000000002 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.284736+0000", 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042411879999999999 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.286680+0000", 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019439069999999999 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530949+0000", 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24426899199999999 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530961+0000", 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1765e-05 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.692 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213982 10.d 10:bb1e392b:::benchmark_data_smithi179_155357_object103486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.280423+0000", 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.404281873000002, 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.249565389, 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213982 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.693 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280423+0000", 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280423+0000", 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280424+0000", 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7599999999999996e-07 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280428+0000", 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0620000000000002e-06 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280445+0000", 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7129000000000002e-05 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280453+0000", 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.208e-06 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280510+0000", 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6900000000000001e-05 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280623+0000", 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000112848 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-03-24T11:37:10.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.280743+0000", 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119781 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.284507+0000", 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003764575 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.284877+0000", 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036966000000000001 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.529978+0000", 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24510062899999999 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.529988+0000", 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0821e-05 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213989 10.12 10:4a136481:::benchmark_data_smithi179_155357_object103493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.285909+0000", 2024-03-24T11:37:10.698 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.398795452999998, 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25090802800000001, 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213989 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.285909+0000", 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.285909+0000", 2024-03-24T11:37:10.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.285910+0000", 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6499999999999998e-07 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.285914+0000", 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6090000000000001e-06 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.285922+0000", 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7649999999999993e-06 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.285927+0000", 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1259999999999997e-06 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.285968+0000", 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0917999999999997e-05 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.286058+0000", 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0529e-05 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2,5", 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.286474+0000", 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041598999999999998 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.295124+0000", 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.008649861 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.295504+0000", 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038038900000000002 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.536669+0000", 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24116443000000001 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.536757+0000", 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7757999999999996e-05 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.536817+0000", 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0887999999999997e-05 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.704 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213978 10.0 10:028c0284:::benchmark_data_smithi179_155357_object103482:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.291978+0000", 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.392727055000002, 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0690288450000001, 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213978 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291978+0000", 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291978+0000", 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291979+0000", 2024-03-24T11:37:10.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.649e-06 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291985+0000", 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1100000000000002e-06 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291999+0000", 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4358999999999999e-05 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292007+0000", 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4989999999999994e-06 2024-03-24T11:37:10.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.531437+0000", 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23942971699999999 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.531447+0000", 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0206e-05 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360935+0000", 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82948748999999999 2024-03-24T11:37:10.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.361007+0000", 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1815000000000003e-05 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213982 10.1d 10:bb1e392b:::benchmark_data_smithi179_155357_object103486:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.292022+0000", 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.392682469, 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1007629329999999, 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.709 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213982 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292022+0000", 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292022+0000", 2024-03-24T11:37:10.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292023+0000", 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9499999999999998e-07 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292025+0000", 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5730000000000002e-06 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292031+0000", 2024-03-24T11:37:10.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1099999999999999e-06 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292036+0000", 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9849999999999997e-06 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530017+0000", 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23798054599999999 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530025+0000", 2024-03-24T11:37:10.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8890000000000002e-06 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.392716+0000", 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86269092599999997 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.392785+0000", 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9509000000000001e-05 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.713 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213995 10.b 10:d077913b:::benchmark_data_smithi179_155357_object103499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.307895+0000", 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.376809911000002, 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.112235474, 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213995 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307895+0000", 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307895+0000", 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307897+0000", 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0269999999999998e-06 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307908+0000", 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1311999999999999e-05 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307930+0000", 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1514000000000001e-05 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307944+0000", 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3890000000000001e-05 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.529700+0000", 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.221756113 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.529712+0000", 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2038000000000001e-05 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.365461+0000", 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83574963800000002 2024-03-24T11:37:10.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.365550+0000", 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8343000000000004e-05 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.365887+0000", 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033723100000000001 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369902+0000", 2024-03-24T11:37:10.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040151600000000003 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.413603+0000", 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043701146000000003 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420032+0000", 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064287880000000004 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420087+0000", 2024-03-24T11:37:10.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4879000000000001e-05 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420130+0000", 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3395e-05 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213996 10.1a 10:5f2e433b:::benchmark_data_smithi179_155357_object103500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.315628+0000", 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.369077273000002, 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1033152850000001, 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.720 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213996 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315628+0000", 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315628+0000", 2024-03-24T11:37:10.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315630+0000", 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9000000000000002e-06 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315640+0000", 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8430000000000005e-06 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315664+0000", 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3883000000000002e-05 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315678+0000", 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4174e-05 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.537200+0000", 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.22152160600000001 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.537210+0000", 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0222e-05 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399490+0000", 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86227972799999997 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399621+0000", 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000130703 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:37:10.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399848+0000", 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022742899999999999 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.400929+0000", 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001080737 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.401373+0000", 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000444719 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418875+0000", 2024-03-24T11:37:10.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017501404000000002 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418909+0000", 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4530999999999999e-05 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418943+0000", 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3405999999999998e-05 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213998 10.1d 10:bc0250f7:::benchmark_data_smithi179_155357_object103502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.334402+0000", 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.350302554999999, 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.084418715, 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213998 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334402+0000", 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334402+0000", 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334405+0000", 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.385e-06 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334415+0000", 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0684e-05 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334434+0000", 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8297000000000001e-05 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334445+0000", 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1430000000000001e-05 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530050+0000", 2024-03-24T11:37:10.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.195604479 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530056+0000", 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4579999999999998e-06 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.393146+0000", 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86308958800000002 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.393218+0000", 2024-03-24T11:37:10.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2366000000000005e-05 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.393413+0000", 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000195145 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.394762+0000", 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013487939999999999 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.395209+0000", 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044712899999999999 2024-03-24T11:37:10.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418726+0000", 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023516927 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418778+0000", 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2154000000000002e-05 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418821+0000", 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2879000000000001e-05 2024-03-24T11:37:10.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214000 10.1b 10:d9e9d0b4:::benchmark_data_smithi179_155357_object103504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.344000+0000", 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.340705057000001, 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0764686059999999, 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214000 2024-03-24T11:37:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344000+0000", 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344000+0000", 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344002+0000", 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6630000000000002e-06 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344013+0000", 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0387000000000001e-05 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344050+0000", 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6708000000000002e-05 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344070+0000", 2024-03-24T11:37:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0489999999999999e-05 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530232+0000", 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18616228600000001 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530240+0000", 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3300000000000001e-06 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.398610+0000", 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86837029200000004 2024-03-24T11:37:10.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.398766+0000", 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015581199999999999 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399018+0000", 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000252031 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.400310+0000", 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012920869999999999 2024-03-24T11:37:10.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.413623+0000", 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013313225999999999 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420372+0000", 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0067491239999999996 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420426+0000", 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3949000000000001e-05 2024-03-24T11:37:10.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420468+0000", 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2221000000000001e-05 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214001 10.a 10:54276112:::benchmark_data_smithi179_155357_object103505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:37:10.740 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.448752+0000", 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.235953160000001, 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96866780900000005, 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214001 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448752+0000", 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448752+0000", 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448754+0000", 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3470000000000001e-06 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448766+0000", 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1715999999999999e-05 2024-03-24T11:37:10.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448798+0000", 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2740000000000002e-05 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448813+0000", 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4953000000000001e-05 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.536914+0000", 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088100303000000005 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.536926+0000", 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2651e-05 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.367204+0000", 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83027731199999999 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.367339+0000", 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013548299999999999 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.367663+0000", 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032424599999999999 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.371158+0000", 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034944820000000001 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.400667+0000", 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.02950906 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.417311+0000", 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016644042000000001 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.417374+0000", 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2748000000000003e-05 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.417419+0000", 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5726000000000002e-05 2024-03-24T11:37:10.746 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215434 10.1c 10.6089207c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 43.359640237999997, 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.998186252, 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.747 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215434 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:37:10.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.99e-07 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325069+0000", 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2710000000000001e-06 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325078+0000", 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3209999999999994e-06 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325084+0000", 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6229999999999999e-06 2024-03-24T11:37:10.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325311+0000", 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022635299999999999 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325319+0000", 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9119999999999998e-06 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323236+0000", 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99791718299999999 2024-03-24T11:37:10.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323251+0000", 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.489e-05 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215438 10.1c 10.5cc036bc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.325397+0000", 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 43.359307835000003, 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99789192500000001, 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.751 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215438 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325397+0000", 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325397+0000", 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325398+0000", 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0999999999999998e-07 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325402+0000", 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8249999999999998e-06 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325456+0000", 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4097000000000001e-05 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325506+0000", 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0275999999999998e-05 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325551+0000", 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4663999999999998e-05 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325557+0000", 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8950000000000001e-06 2024-03-24T11:37:10.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323281+0000", 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99772327199999999 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323289+0000", 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1859999999999992e-06 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.755 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215480 10.1c 10.e4fdbf3c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.354574+0000", 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 43.330130662000002, 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96874779600000005, 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215480 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354574+0000", 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354574+0000", 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354575+0000", 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1099999999999995e-07 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354578+0000", 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3790000000000002e-06 2024-03-24T11:37:10.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354585+0000", 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6420000000000001e-06 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354590+0000", 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4840000000000003e-06 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354632+0000", 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1233999999999999e-05 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354637+0000", 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1109999999999999e-06 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323314+0000", 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96867741799999996 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323322+0000", 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9170000000000006e-06 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215504 10.1c 10.f3f889bc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.370318+0000", 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 43.314386376000002, 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95304131700000005, 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.760 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215504 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370318+0000", 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370318+0000", 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370319+0000", 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8599999999999998e-07 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370323+0000", 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9589999999999997e-06 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370332+0000", 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2799999999999992e-06 2024-03-24T11:37:10.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370339+0000", 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7730000000000002e-06 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370386+0000", 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6835999999999997e-05 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370392+0000", 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5620000000000003e-06 2024-03-24T11:37:10.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323352+0000", 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95296023100000005 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323360+0000", 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9899999999999997e-06 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215517 10.1c 10.9bdc429c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:37:10.764 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.388053+0000", 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 43.296652020000003, 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.93534938300000003, 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215517 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388053+0000", 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388053+0000", 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388053+0000", 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9599999999999999e-07 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388057+0000", 2024-03-24T11:37:10.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7869999999999999e-06 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388067+0000", 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5419999999999994e-06 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388073+0000", 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8869999999999997e-06 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388120+0000", 2024-03-24T11:37:10.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6981999999999998e-05 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.388128+0000", 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4449999999999996e-06 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323393+0000", 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.93526488500000005 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323402+0000", 2024-03-24T11:37:10.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1589999999999999e-06 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215439 10.1f 10:fa4c0bae:::benchmark_data_smithi179_155357_object104943:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318381+0000", 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.366324159999998, 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.125454446, 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215439 2024-03-24T11:37:10.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318381+0000", 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318381+0000", 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318383+0000", 2024-03-24T11:37:10.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-06 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318390+0000", 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318410+0000", 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9894e-05 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318423+0000", 2024-03-24T11:37:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3715e-05 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318532+0000", 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108372 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318541+0000", 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6039999999999999e-06 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321920+0000", 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033789620000000001 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321930+0000", 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7510000000000003e-06 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434157+0000", 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1122265600000001 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434437+0000", 2024-03-24T11:37:10.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028081500000000001 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443751+0000", 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0093135800000000001 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443835+0000", 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4095e-05 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215473 10.1f 10:f8f7c4ee:::benchmark_data_smithi179_155357_object104977:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.366249259, 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1260989669999999, 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215473 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3799999999999997e-07 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318460+0000", 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7479999999999999e-06 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318610+0000", 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000150109 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318621+0000", 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0552e-05 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318797+0000", 2024-03-24T11:37:10.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000175892 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318802+0000", 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7950000000000004e-06 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321977+0000", 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003174718 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321983+0000", 2024-03-24T11:37:10.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4370000000000004e-06 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434809+0000", 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1128255279999999 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434952+0000", 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014316599999999999 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.435226+0000", 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000273773 2024-03-24T11:37:10.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439135+0000", 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039092069999999996 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443122+0000", 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039872429999999997 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443876+0000", 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000753132 2024-03-24T11:37:10.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443980+0000", 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104318 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444253+0000", 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027285599999999998 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444360+0000", 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107435 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444555+0000", 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019442 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215489 10.1f 10:f9e29235:::benchmark_data_smithi179_155357_object104993:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318646+0000", 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.366058475000003, 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1283527250000001, 2024-03-24T11:37:10.782 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215489 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318646+0000", 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318646+0000", 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318647+0000", 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318652+0000", 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8199999999999996e-06 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318662+0000", 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9879999999999999e-06 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318669+0000", 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3379999999999999e-06 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318832+0000", 2024-03-24T11:37:10.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000162351 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318837+0000", 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0080000000000002e-06 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322008+0000", 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031713230000000002 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322015+0000", 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6830000000000002e-06 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.436372+0000", 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1143569840000001 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.436486+0000", 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011397599999999999 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.436692+0000", 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020638399999999999 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443183+0000", 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064913000000000002 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444114+0000", 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00093082500000000003 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444156+0000", 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1903999999999997e-05 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444629+0000", 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047258800000000002 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446765+0000", 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021363089999999999 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446859+0000", 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4137999999999999e-05 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446999+0000", 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000139896 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.790 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215498 10.1f 10:faa9498f:::benchmark_data_smithi179_155357_object105002:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318681+0000", 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.366023863000002, 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1285948139999999, 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215498 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318681+0000", 2024-03-24T11:37:10.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318681+0000", 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318682+0000", 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2699999999999999e-07 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318685+0000", 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5200000000000002e-06 2024-03-24T11:37:10.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318693+0000", 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6020000000000002e-06 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318699+0000", 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8089999999999998e-06 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318863+0000", 2024-03-24T11:37:10.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016406799999999999 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318867+0000", 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8369999999999996e-06 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322038+0000", 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031707179999999999 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322044+0000", 2024-03-24T11:37:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1639999999999998e-06 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.437844+0000", 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1157992670000001 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.437969+0000", 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012585 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.438160+0000", 2024-03-24T11:37:10.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000190197 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444029+0000", 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058697849999999998 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444234+0000", 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020496199999999999 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444612+0000", 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037775500000000002 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444636+0000", 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3785e-05 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447057+0000", 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024206219999999999 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447129+0000", 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2862999999999997e-05 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447276+0000", 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014638300000000001 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215505 10.1f 10:fb4d5417:::benchmark_data_smithi179_155357_object105009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.365997835999998, 2024-03-24T11:37:10.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1288068360000001, 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215505 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:37:10.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2300000000000002e-07 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318710+0000", 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0680000000000001e-06 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:37:10.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318718+0000", 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5120000000000002e-06 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318730+0000", 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2084e-05 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318893+0000", 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016259799999999999 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318897+0000", 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6e-06 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322066+0000", 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031691530000000001 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322073+0000", 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4529999999999999e-06 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:37:10.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439183+0000", 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1171100140000001 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439307+0000", 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012411799999999999 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439496+0000", 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000189333 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444076+0000", 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004579301 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444641+0000", 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056571499999999995 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444687+0000", 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5529999999999999e-05 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446602+0000", 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001915017 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447324+0000", 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00072233799999999999 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447383+0000", 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9181999999999997e-05 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:37:10.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447514+0000", 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013039700000000001 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:37:10.807 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:37:10.839 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:10.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:10.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:10.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:10.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:11.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:11.124 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:11.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:11.138 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 8 16 19697 19681 153.617 138.438 0.00357482 0.0065029 2024-03-24T11:37:11.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:11.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:11.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:11.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:11.393 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:11.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:11.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:11.501 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:11.547 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:11.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:11.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:11.679 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:11.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:11.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:11.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:11.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:11.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:11 smithi179 ceph-mon[29608]: pgmap v842: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 932 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s rd, 116 MiB/s wr, 1.86k op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:37:11.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:11 smithi179 ceph-mon[37907]: pgmap v842: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 932 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s rd, 116 MiB/s wr, 1.86k op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:37:11.889 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:11 smithi181 ceph-mon[33998]: pgmap v842: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 932 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s rd, 116 MiB/s wr, 1.86k op/s; 0 B/s, 7 objects/s recovering 2024-03-24T11:37:11.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:11.949 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:11.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:12.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:12.086 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:12.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:12.138 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 9 16 21742 21726 150.75 127.812 0.00628353 0.00662707 2024-03-24T11:37:12.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:12.223 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:12.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:12.324 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:12.367 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:12.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:12.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:12.500 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:12.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:12.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:12.642 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:12.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:12.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:12.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T11:37:12.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:12.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:12.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:12.914 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:12.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:13.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:13.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:13.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:13.135 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:13.139 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 10 15 23692 23677 147.87 121.938 0.0117214 0.00675658 2024-03-24T11:37:13.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:13.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:13.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:13.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:13.330 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:13.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:13.432 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:13.470 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:13.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:13.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:13.614 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:13.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:13 smithi179 ceph-mon[29608]: pgmap v843: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 932 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T11:37:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:13 smithi179 ceph-mon[37907]: pgmap v843: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 932 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T11:37:13.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:13.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:13.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:13.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:13 smithi181 ceph-mon[33998]: pgmap v843: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 932 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T11:37:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:13.904 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:13.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:14.004 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:14.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:14.052 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:14.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:14.139 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 11 15 25507 25492 144.739 113.438 0.00919016 0.00690308 2024-03-24T11:37:14.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:14.185 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:14.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:14.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:14.342 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:14.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:14.378 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:37:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:37:14.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:14.479 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:14.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:14.581 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:14.625 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:14.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:14.637 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:37:14.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:14.765 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:14.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:14.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:14.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:14 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2974322219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:14.886 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:14 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:14.890 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:14 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2974322219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:14.890 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:14 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:14.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:14 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2974322219' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:14.890 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:14 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:14.912 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:14.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:15.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:15.055 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:15.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:15.063 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:37:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:37:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-03-24T11:37:15.064 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:37:15.139 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 12 15 27402 27387 142.547 118.438 0.00712157 0.00700823 2024-03-24T11:37:15.157 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:15.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:15.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:15.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:37:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:37:15.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:15.343 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:15.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:15.444 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:15.488 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:15.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:15.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:15.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:15.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:15.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:15.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:15.759 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:15.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:15.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:15.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:15.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:15 smithi181 ceph-mon[33998]: pgmap v844: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-24T11:37:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:15 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:15 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:37:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:15 smithi181 ceph-mon[33998]: osdmap e394: 8 total, 8 up, 7 in 2024-03-24T11:37:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[29608]: pgmap v844: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-24T11:37:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:37:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[29608]: osdmap e394: 8 total, 8 up, 7 in 2024-03-24T11:37:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[37907]: pgmap v844: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-24T11:37:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:37:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:15 smithi179 ceph-mon[37907]: osdmap e394: 8 total, 8 up, 7 in 2024-03-24T11:37:15.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T11:37:15.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:16.037 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:16.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:16.140 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 13 16 29312 29296 140.76 119.312 0.00262452 0.0070976 2024-03-24T11:37:16.172 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:16.274 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:16.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:16.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:16.346 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:16.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:16.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.462 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:16.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:16.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:16.595 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:16.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:16.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:16.829 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:16.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:16.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:16.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:16.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:17.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:17.117 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:17.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:17.139 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 14 16 30869 30853 137.658 97.3125 0.00629678 0.00725845 2024-03-24T11:37:17.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:17.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:17.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:17.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:17.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:17.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:17.483 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:17.524 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:17.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:17.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:17.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:17.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:17.760 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:17.799 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:17.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:17.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:17.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:17 smithi181 ceph-mon[33998]: pgmap v846: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-24T11:37:17.937 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:17.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:17 smithi179 ceph-mon[29608]: pgmap v846: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-24T11:37:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:17 smithi179 ceph-mon[37907]: pgmap v846: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-24T11:37:18.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:18.078 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:18.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:18.139 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 15 16 32582 32566 135.618 107.062 0.00711077 0.00736901 2024-03-24T11:37:18.181 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:18.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:18.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:18.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:18.350 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:18.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:18.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:18.480 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:18.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:18.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:18.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:18.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:18.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:18.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:18.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:18.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:18.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:18.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:18.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:18 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:37:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:18 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:37:18.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:18 smithi181 ceph-mon[33998]: osdmap e395: 8 total, 8 up, 7 in 2024-03-24T11:37:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:18 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:37:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:18 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:37:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:18 smithi179 ceph-mon[29608]: osdmap e395: 8 total, 8 up, 7 in 2024-03-24T11:37:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:18 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:37:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:18 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:37:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:18 smithi179 ceph-mon[37907]: osdmap e395: 8 total, 8 up, 7 in 2024-03-24T11:37:18.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:19.015 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:19.117 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:19.139 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 16 16 34263 34247 133.707 105.062 0.0115617 0.00747224 2024-03-24T11:37:19.157 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:19.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:19.296 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:19.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.436 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:19.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:19.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:19.567 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:19.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.700 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:19.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:19.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:19.836 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:19.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:19 smithi181 ceph-mon[33998]: pgmap v848: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T11:37:19.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:19.862 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:19.934 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:37:19.934 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:37:19.935 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:37:19.935 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:37:19.935 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:37:19.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:19.974 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:19.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:20.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:20.116 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:20.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:20.140 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 17 16 35881 35865 131.791 101.125 0.00497096 0.00758213 2024-03-24T11:37:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:19 smithi179 ceph-mon[29608]: pgmap v848: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T11:37:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:20.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:19 smithi179 ceph-mon[37907]: pgmap v848: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-24T11:37:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:20.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:20.260 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:20.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:20.362 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:20.391 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:20.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:20.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:20.545 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:20.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:20.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:20.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:20.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:20.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:20.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:20.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:20.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T11:37:20.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:20.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:20.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:21.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:21.090 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:21.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:21.141 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 18 16 37561 37545 130.296 105 0.004552 0.00766743 2024-03-24T11:37:21.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:21.232 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:21.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:21.235 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:37:21.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:21.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:21.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:21.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:21.511 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:21.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:21.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:21.654 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:21.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:21.756 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:21.796 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:21.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:21.865 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:37:21.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:21.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:21 smithi181 ceph-mon[33998]: pgmap v849: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-03-24T11:37:21.922 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:21.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:22.024 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:22.075 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:22.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:22.142 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 19 16 39027 39011 128.253 91.625 0.00673477 0.00779008 2024-03-24T11:37:22.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:22.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:21 smithi179 ceph-mon[29608]: pgmap v849: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-03-24T11:37:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:21 smithi179 ceph-mon[37907]: pgmap v849: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-03-24T11:37:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:22.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:22.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:22.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:22.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:22.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:22.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:22.475 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:22.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:22.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:22.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:22.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:22.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:22.754 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:22.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:22.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:22 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:37:22.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:22 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:37:22.832 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:22 smithi179 ceph-mon[29608]: osdmap e396: 8 total, 8 up, 7 in 2024-03-24T11:37:22.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:22 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:37:22.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:22 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:37:22.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:22 smithi179 ceph-mon[37907]: osdmap e396: 8 total, 8 up, 7 in 2024-03-24T11:37:22.855 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:22.896 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:22.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:22.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:23.038 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:23.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:23.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:22 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:37:23.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:22 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:37:23.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:22 smithi181 ceph-mon[33998]: osdmap e396: 8 total, 8 up, 7 in 2024-03-24T11:37:23.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:23.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:37:23.143490+0000 min lat: 0.00109547 max lat: 0.0912904 avg lat: 0.00789841 2024-03-24T11:37:23.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:37:23.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 20 16 40517 40501 126.493 93.125 0.018855 0.00789841 2024-03-24T11:37:23.173 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:23.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:23.208 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:37:23.254 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:37:23.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:23.297 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:37:23.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:23.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:23.340 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:37:23.380 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:37:23.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:23.421 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:37:23.432 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:23.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:23.470 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:37:23.490 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:23.517 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:37:23.533 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:37:23.538 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:37:23.595 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:23.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:23.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:23.735 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:23.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:23.768 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:37:23.768 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:37:23.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:23.868 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:23.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:23 smithi179 ceph-mon[29608]: pgmap v851: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 657 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-24T11:37:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/311329720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:23 smithi179 ceph-mon[37907]: pgmap v851: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 657 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-24T11:37:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/311329720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:23.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:23.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:24.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:24.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:23 smithi181 ceph-mon[33998]: pgmap v851: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 657 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-03-24T11:37:24.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/311329720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:24.132 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:24.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 21 16 41766 41750 124.188 78.0625 0.0059115 0.00804589 2024-03-24T11:37:24.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:24.265 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:24.319 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T11:37:24.367 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:24.398 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:24.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:37:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:37:24.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:24.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T11:37:24.542 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:24.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:24.673 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:24.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:24.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:24.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2510477525' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:37:24.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:37:24.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:37:24.817 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[29608]: osdmap e397: 8 total, 8 up, 7 in 2024-03-24T11:37:24.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2510477525' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:37:24.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:37:24.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:37:24.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:24.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:24.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:25.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:25.074 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:25.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:25.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:25.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 22 16 43286 43270 122.861 95 0.00484242 0.00813378 2024-03-24T11:37:25.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:24 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2510477525' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:37:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:24 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:37:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:24 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:37:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:24 smithi181 ceph-mon[33998]: osdmap e397: 8 total, 8 up, 7 in 2024-03-24T11:37:25.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:25.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:24 smithi179 ceph-mon[37907]: osdmap e397: 8 total, 8 up, 7 in 2024-03-24T11:37:25.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:37:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:37:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:37:25.218 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:25.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:25.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:25.351 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:25.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:25.451 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T11:37:25.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:25.500 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:25.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:25.602 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:25.668 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:25.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:25.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:25.802 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:25.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:25.904 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:25.945 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:25.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:26.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:26.076 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:26.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:26.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 23 16 44720 44704 121.415 89.625 0.00522045 0.00823056 2024-03-24T11:37:26.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:26.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:25 smithi181 ceph-mon[33998]: pgmap v853: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-24T11:37:26.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:25 smithi179 ceph-mon[29608]: pgmap v853: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-24T11:37:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:25 smithi179 ceph-mon[37907]: pgmap v853: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-24T11:37:26.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:26.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:26.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:26.353 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:26.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:26.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:26.484 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:26.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:26.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:26.616 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:26.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:26.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:26.763 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:26.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:26.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:26.915 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:26.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:27.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:27.059 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:27.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:27.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 24 16 46177 46161 120.151 91.0625 0.00880762 0.00831613 2024-03-24T11:37:27.153 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:27.153 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:27.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:27.187 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:27.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:27.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:27.318 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:27.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:27.420 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:27.465 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:27.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:27.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:27.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:27.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:27.705 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:27.747 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:27.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:27.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:27.877 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:27.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:27.978 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:28.015 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:28.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:28.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 25 16 47513 47497 118.685 83.5 0.0138407 0.00841927 2024-03-24T11:37:28.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:28.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:27 smithi181 ceph-mon[33998]: pgmap v854: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:37:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:27 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:37:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:27 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:37:28.200 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:27 smithi179 ceph-mon[29608]: pgmap v854: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:37:28.201 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:27 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:37:28.201 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:27 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:37:28.201 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:27 smithi179 ceph-mon[37907]: pgmap v854: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-24T11:37:28.201 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:27 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:37:28.202 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:27 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:37:28.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:28.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:28.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:28.415 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:28.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:28.553 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:28.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:28.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:28.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:37:28.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.842 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:28.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:28.861 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:37:28.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:28.975 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:28.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:29.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:29.126 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:29.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:29.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 26 16 48934 48918 117.535 88.8125 0.00707826 0.00850288 2024-03-24T11:37:29.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:28 smithi181 ceph-mon[33998]: osdmap e398: 8 total, 8 up, 7 in 2024-03-24T11:37:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1918926384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:37:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:28 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:37:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:28 smithi179 ceph-mon[29608]: osdmap e398: 8 total, 8 up, 7 in 2024-03-24T11:37:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1918926384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:37:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:28 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:37:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:28 smithi179 ceph-mon[37907]: osdmap e398: 8 total, 8 up, 7 in 2024-03-24T11:37:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1918926384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:37:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:28 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:37:29.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:29.277 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:29.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:29.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:29.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:29.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:29.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:29.536 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:29.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:29.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:29.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:29.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:29.766 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:29.791 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:29.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:29.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:29.931 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:29.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[29608]: pgmap v856: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.5 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T11:37:29.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:37:29.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[29608]: osdmap e399: 8 total, 8 up, 7 in 2024-03-24T11:37:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[37907]: pgmap v856: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.5 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T11:37:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:37:29.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:29 smithi179 ceph-mon[37907]: osdmap e399: 8 total, 8 up, 7 in 2024-03-24T11:37:30.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:30.077 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:30.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:30.143 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 27 16 50284 50268 116.307 84.375 0.00527088 0.00859287 2024-03-24T11:37:30.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:29 smithi181 ceph-mon[33998]: pgmap v856: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.5 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T11:37:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:29 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:37:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:29 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:37:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:29 smithi181 ceph-mon[33998]: osdmap e399: 8 total, 8 up, 7 in 2024-03-24T11:37:30.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:30.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:30.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:30.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:30.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:30.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:30.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:30.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:30.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:30.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:30.615 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:30.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:30.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:30.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:30.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:30.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:30.892 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:30.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:30.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.031 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:31.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:31.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:31.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 28 16 51520 51504 114.913 77.25 0.00502956 0.00867183 2024-03-24T11:37:31.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:31.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:31.291 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:31.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:31.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:31.526 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:31.556 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:31.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:31.699 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:31.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.842 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:31.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:31.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:31.967 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:31.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:32.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:32.100 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:32.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:32.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 29 16 52780 52764 113.666 78.75 0.00401308 0.00879157 2024-03-24T11:37:32.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:31 smithi181 ceph-mon[33998]: pgmap v858: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 837 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-24T11:37:32.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:32.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:31 smithi179 ceph-mon[29608]: pgmap v858: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 837 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-24T11:37:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:31 smithi179 ceph-mon[37907]: pgmap v858: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 837 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-24T11:37:32.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:32.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:32.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:32.373 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:32.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:32.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:32.504 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:32.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:32.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:32.635 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:32.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:32.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:32.779 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:32.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:32.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:32.908 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:32.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:33.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:33.042 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:33.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:33.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 30 16 54182 54166 112.798 87.625 0.00666251 0.00885929 2024-03-24T11:37:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:37:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:37:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:37:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:37:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]: dispatch 2024-03-24T11:37:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]: dispatch 2024-03-24T11:37:33.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2024-03-24T11:37:33.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]: dispatch 2024-03-24T11:37:33.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]: dispatch 2024-03-24T11:37:33.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:32 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2024-03-24T11:37:33.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:37:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:37:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:37:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:37:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]: dispatch 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]: dispatch 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]: dispatch 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]: dispatch 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:37:33.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]: dispatch 2024-03-24T11:37:33.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]: dispatch 2024-03-24T11:37:33.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2024-03-24T11:37:33.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]: dispatch 2024-03-24T11:37:33.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]: dispatch 2024-03-24T11:37:33.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:32 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2024-03-24T11:37:33.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:33.318 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:33.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:33.449 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:33.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:33.580 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:33.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:33.711 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:33.812 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:33.848 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:33.950 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: pgmap v859: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:37:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:37:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:37:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]': finished 2024-03-24T11:37:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]': finished 2024-03-24T11:37:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2024-03-24T11:37:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: osdmap e400: 8 total, 8 up, 7 in 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: 4.0 scrub starts 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: 4.0 scrub ok 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: pgmap v859: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]': finished 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]': finished 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: osdmap e400: 8 total, 8 up, 7 in 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: 4.0 scrub starts 2024-03-24T11:37:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: 4.0 scrub ok 2024-03-24T11:37:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:33.991 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:33.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:34.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:34.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:34.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: pgmap v859: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [3, 5]}]': finished 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 0]}]': finished 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: osdmap e400: 8 total, 8 up, 7 in 2024-03-24T11:37:34.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: 4.0 scrub starts 2024-03-24T11:37:34.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: 4.0 scrub ok 2024-03-24T11:37:34.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:34.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 31 16 55562 55546 111.941 86.25 0.0268249 0.00892699 2024-03-24T11:37:34.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:34.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:34.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:34.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:34.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:34.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:34.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:37:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:37:34.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:34.530 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:34.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:34.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:34.663 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:34.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:34.726 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:37:34.726 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:37:34.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:34.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:34.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:34.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:34.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:34.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:34.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:35.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:35.058 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:35.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:35.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 32 16 56781 56765 110.824 76.1875 0.00845421 0.00901813 2024-03-24T11:37:35.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:34 smithi181 ceph-mon[33998]: osdmap e401: 8 total, 8 up, 7 in 2024-03-24T11:37:35.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:35.200 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:35.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:35.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:34 smithi179 ceph-mon[29608]: osdmap e401: 8 total, 8 up, 7 in 2024-03-24T11:37:35.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:37:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:37:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:34 smithi179 ceph-mon[37907]: osdmap e401: 8 total, 8 up, 7 in 2024-03-24T11:37:35.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:35.341 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:35.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:35.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:35.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:35.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:35.574 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:35.607 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:35.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:35.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:35.754 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:35.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:35.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:35.871 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:35.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T11:37:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:35.918 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:35.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:36.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:36.079 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:36.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:36.107 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:36.107 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: pgmap v862: 158 pgs: 3 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:37:36.107 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:37:36.107 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:37:36.107 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:37:36.107 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:37:36.107 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: pgmap v862: 158 pgs: 3 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:37:36.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:35 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:37:36.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 33 16 58057 58041 109.883 79.75 0.0172738 0.00909611 2024-03-24T11:37:36.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: pgmap v862: 158 pgs: 3 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-24T11:37:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:37:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:37:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:37:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:37:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:37:36.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:35 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:37:36.174 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:37:36.180 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:36.208 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:36.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:36.310 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:36.386 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:36.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:36.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:36.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:36.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:36.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:36.645 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:36.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:36.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:36.776 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:36.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:36.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:36.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:36.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:37.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:37.060 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:37.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:37.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 34 16 59225 59209 108.798 73 0.00592606 0.00918647 2024-03-24T11:37:37.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:37.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:37.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:37.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:37.325 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:37.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:37.333 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T11:37:37.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:37.455 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:37.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:37.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:37.600 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:37.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: 3.2 scrub starts 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: 3.2 scrub ok 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: osdmap e402: 8 total, 8 up, 7 in 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: pgmap v864: 158 pgs: 3 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T11:37:37.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4049924031' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:37:37.654 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:37 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:37:37.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:37:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:37:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:37:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: 3.2 scrub starts 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: 3.2 scrub ok 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: osdmap e402: 8 total, 8 up, 7 in 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: pgmap v864: 158 pgs: 3 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4049924031' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:37:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: 3.2 scrub starts 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: 3.2 scrub ok 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: osdmap e402: 8 total, 8 up, 7 in 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: pgmap v864: 158 pgs: 3 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4049924031' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:37:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:37 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:37:37.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:37.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:37.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:37.873 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:37.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:37.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:38.013 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:38.115 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:38.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 35 16 60526 60510 108.013 81.3125 0.0131608 0.00925224 2024-03-24T11:37:38.152 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:38.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:38.283 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:38.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:38.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:38.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.540 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:38.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:38.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:38.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:37:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:37:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:37:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:37:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:37:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: osdmap e403: 8 total, 8 up, 7 in 2024-03-24T11:37:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:37:38.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:38 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:37:38.670 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:37:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:37:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:37:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: osdmap e403: 8 total, 8 up, 7 in 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: osdmap e403: 8 total, 8 up, 7 in 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:37:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:38 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:37:38.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:38.800 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:38.902 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:38.942 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:38.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:39.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:39.083 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:39.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:39.144 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 36 15 61599 61584 106.877 67.125 0.0192204 0.00935121 2024-03-24T11:37:39.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:39.218 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:39.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:39.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:39.344 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:39.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:39.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:39.488 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:39.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:39.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T11:37:39.583 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:39.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:39.610 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:37:39.610 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:37:39.610 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:37:39.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:37:39.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[37907]: pgmap v866: 158 pgs: 2 peering, 1 active+remapped+backfill_toofull, 155 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.8 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 64 B/s, 1 objects/s recovering 2024-03-24T11:37:39.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:37:39.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:37:39.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:37:39.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:37:39.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:37:39.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:37:39.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[29608]: pgmap v866: 158 pgs: 2 peering, 1 active+remapped+backfill_toofull, 155 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.8 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 64 B/s, 1 objects/s recovering 2024-03-24T11:37:39.613 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:37:39.614 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:39 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:37:39.619 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:39.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:39.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:39 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:37:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:39 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:37:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:39 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:37:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:39 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:37:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:39 smithi181 ceph-mon[33998]: pgmap v866: 158 pgs: 2 peering, 1 active+remapped+backfill_toofull, 155 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.8 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 64 B/s, 1 objects/s recovering 2024-03-24T11:37:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:39 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:37:39.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:39 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:37:39.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:39.776 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:39.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:39.792 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:39.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:39.930 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:39.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:40.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:40.061 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:40.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 37 16 62655 62639 105.771 65.9375 0.0117628 0.00944746 2024-03-24T11:37:40.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:40.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:40.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:40.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:40.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:40.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:40.534 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:37:40.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:40.573 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:40 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:37:40.573 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:40 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:37:40.573 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:40 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:37:40.573 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:40 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:37:40.581 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:40.607 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:37:40.649 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:37:40.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:40 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:37:40.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:40 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:37:40.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:40.698 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:37:40.724 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:40.734 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:37:40.772 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:37:40.811 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:37:40.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:40.854 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:37:40.865 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:40.865 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:37:40.865 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:37:40.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:37:40.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:37:40.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:37:40.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:37:40.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:37:40.866 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:37:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:40.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:40.992 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:40.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:41.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:41.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:41.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:41.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 38 16 63724 63708 104.746 66.8125 0.00548476 0.00954172 2024-03-24T11:37:41.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:41.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:41.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:41.388 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:41.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:37:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:37:41.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[29608]: pgmap v867: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 781 B/s rd, 78 MiB/s wr, 1.25k op/s; 71 B/s, 2 objects/s recovering 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3967612239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[37907]: pgmap v867: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 781 B/s rd, 78 MiB/s wr, 1.25k op/s; 71 B/s, 2 objects/s recovering 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3967612239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:37:41.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:41 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:37:41.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:41.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:41.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:41.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:41.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:41.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:41 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:37:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:41 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:37:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:41 smithi181 ceph-mon[33998]: pgmap v867: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 781 B/s rd, 78 MiB/s wr, 1.25k op/s; 71 B/s, 2 objects/s recovering 2024-03-24T11:37:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3967612239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:41 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:37:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:41 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:37:41.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:41.802 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:41.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:41.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:41.932 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:41.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:42.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:42.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:42.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:42.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 39 16 64867 64851 103.892 71.4375 0.013902 0.00961949 2024-03-24T11:37:42.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:42.198 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:42.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:42.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:42.341 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:42.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:42.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:42.473 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:42.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:42.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:42.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:42.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:42 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:37:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:42 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:37:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:42 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:37:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:42 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:37:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:42 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:37:42.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:42 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:37:42.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:42.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:37:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:37:42.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:37:42.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:37:42.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:42 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:37:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:42.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:42.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:42.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:42.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:42.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:42.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:43.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:43.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:43.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:37:43.145679+0000 min lat: 0.00109547 max lat: 0.156982 avg lat: 0.00969168 2024-03-24T11:37:43.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:37:43.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 40 16 66037 66021 103.123 73.125 0.0144436 0.00969168 2024-03-24T11:37:43.155 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:43.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:43.289 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:43.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:43.420 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:43.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:43.561 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:43 smithi181 ceph-mon[33998]: 1.4 scrub starts 2024-03-24T11:37:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:43 smithi181 ceph-mon[33998]: 1.4 scrub ok 2024-03-24T11:37:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:43 smithi181 ceph-mon[33998]: 6.1c scrub starts 2024-03-24T11:37:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:43 smithi181 ceph-mon[33998]: 6.1c scrub ok 2024-03-24T11:37:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:43 smithi181 ceph-mon[33998]: pgmap v868: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 58 B/s, 1 objects/s recovering 2024-03-24T11:37:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:43 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:37:43.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:43 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:37:43.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:43.663 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[29608]: 1.4 scrub starts 2024-03-24T11:37:43.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[29608]: 1.4 scrub ok 2024-03-24T11:37:43.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[29608]: 6.1c scrub starts 2024-03-24T11:37:43.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[29608]: 6.1c scrub ok 2024-03-24T11:37:43.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[29608]: pgmap v868: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 58 B/s, 1 objects/s recovering 2024-03-24T11:37:43.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:37:43.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:37:43.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[37907]: 1.4 scrub starts 2024-03-24T11:37:43.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[37907]: 1.4 scrub ok 2024-03-24T11:37:43.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[37907]: 6.1c scrub starts 2024-03-24T11:37:43.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[37907]: 6.1c scrub ok 2024-03-24T11:37:43.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[37907]: pgmap v868: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 58 B/s, 1 objects/s recovering 2024-03-24T11:37:43.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:37:43.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:43 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:37:43.702 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:43.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.845 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:43.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:43.945 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:43.984 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:43.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:44.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:44.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:44.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:44.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 41 16 67290 67274 102.517 78.3125 0.0150791 0.00974923 2024-03-24T11:37:44.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:44.251 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:44.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:44.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:44.393 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:44.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:37:44.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:37:44.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:37:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:37:44.495 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:44.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:44.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:44.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:44.668 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:44.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:37:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:37:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:37:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:37:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:37:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:37:44.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:44 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:37:44.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:44.818 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:44.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:44.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:44.965 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:44.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:45.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:45.096 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:45.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:45.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 42 16 68487 68471 101.858 74.8125 0.00558916 0.0098126 2024-03-24T11:37:45.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:45.226 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:45.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:45.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:45.358 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:37:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:37:45.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:45.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:45.461 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:45.499 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:45.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:45.600 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:45.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:45.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:45.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:45 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:37:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:45 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:37:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:45 smithi181 ceph-mon[33998]: pgmap v869: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 57 B/s, 1 objects/s recovering 2024-03-24T11:37:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:45 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:37:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:45 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:37:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:37:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:37:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[29608]: pgmap v869: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 57 B/s, 1 objects/s recovering 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[37907]: pgmap v869: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 57 B/s, 1 objects/s recovering 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:37:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:45 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:37:45.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:45.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:45.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:45.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:45.902 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:45.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:46.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:46.033 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:46.136 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:46.146 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 43 15 69649 69634 101.179 72.6875 0.0216859 0.00987689 2024-03-24T11:37:46.175 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:46.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:46.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:46.305 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:46.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:46.437 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:46.539 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.597 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:46.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:46.648 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:37:46.648 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:37:46.648 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:37:46.648 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:37:46.648 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:37:46.648 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:37:46.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:37:46.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:46 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:37:46.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:46 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:37:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:46 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:37:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:46 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:37:46.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:46 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:37:46.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:46.727 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:46.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:46.863 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:46.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:46.997 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:46.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:47.099 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:47.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:47.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:47.145 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 44 16 70770 70754 100.471 70 0.0150455 0.00994797 2024-03-24T11:37:47.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:47.277 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:47.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:47.344 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:47.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:47.389 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:37:47.389 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:37:47.389 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:37:47.390 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:37:47.390 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:37:47.405 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:47.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:37:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:37:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[29608]: pgmap v870: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 46 B/s, 1 objects/s recovering 2024-03-24T11:37:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:37:47.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:37:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:37:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:37:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[37907]: pgmap v870: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 46 B/s, 1 objects/s recovering 2024-03-24T11:37:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:37:47.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:47 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:37:47.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:47.538 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:47.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:47.639 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:47.685 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:47.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:47.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:47.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:37:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:47.831 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:47.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:47 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:37:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:47 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:37:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:47 smithi181 ceph-mon[33998]: pgmap v870: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 46 B/s, 1 objects/s recovering 2024-03-24T11:37:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:47 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:37:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:47 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:37:47.933 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:47.961 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:47.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:48.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:48.089 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:48.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:48.146 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 45 16 71786 71770 99.6478 63.5 0.0111797 0.0100304 2024-03-24T11:37:48.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:48.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:48.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:48.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:48.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:48.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:48.429 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:37:48.465 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:48.507 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:48.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:48.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:48.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:48.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2504469105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:37:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:37:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2504469105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:37:48.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:48 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:37:48.747 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:48.792 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:48.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:48.893 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:48.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:48 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2504469105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:48.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:48 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:37:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:48 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:37:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:48 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:37:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:48.923 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:48.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:49.024 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:49.066 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:49.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:49.146 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 46 16 72881 72865 98.9693 68.4375 0.0105653 0.010099 2024-03-24T11:37:49.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:49.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:49.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:49.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:49.333 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:49.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:49.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:49.471 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:49.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:49.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:49.615 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:49.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:37:49.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: osdmap e404: 8 total, 8 up, 7 in 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: pgmap v872: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 10 B/s, 0 objects/s recovering 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[29608]: osdmap e405: 8 total, 8 up, 7 in 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:37:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:37:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: osdmap e404: 8 total, 8 up, 7 in 2024-03-24T11:37:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: pgmap v872: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 10 B/s, 0 objects/s recovering 2024-03-24T11:37:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:37:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:37:49.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:49 smithi179 ceph-mon[37907]: osdmap e405: 8 total, 8 up, 7 in 2024-03-24T11:37:49.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:49.755 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:49.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:49.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:49.895 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:49.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: osdmap e404: 8 total, 8 up, 7 in 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: pgmap v872: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 10 B/s, 0 objects/s recovering 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:37:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:49 smithi181 ceph-mon[33998]: osdmap e405: 8 total, 8 up, 7 in 2024-03-24T11:37:49.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:50.039 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:50.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:50.146 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 47 16 74093 74077 98.4753 75.75 0.00630535 0.0101494 2024-03-24T11:37:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.176 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:50.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:50.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.324 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:50.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:50.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:50.451 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:50.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T11:37:50.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:50.580 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:50.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:50 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:37:50.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:50 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:37:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:50 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:37:50.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:50 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:37:50.712 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:50.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:50.843 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:50.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:50 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:37:50.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:50 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:37:50.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:50.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:50.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:51.077 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:51.105 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:51.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:51.146 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 48 16 75214 75198 97.8834 70.0625 0.00642156 0.0102111 2024-03-24T11:37:51.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:51.244 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:51.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:51.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:51.390 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:51.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:51.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:51.530 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:51.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:51.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:51.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:51.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:51 smithi179 ceph-mon[29608]: pgmap v874: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-24T11:37:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:51 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:37:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:51 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:37:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:51 smithi179 ceph-mon[37907]: pgmap v874: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-24T11:37:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:51 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:37:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:51 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:37:51.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:51.805 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:51.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:51 smithi181 ceph-mon[33998]: pgmap v874: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-24T11:37:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:51 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:37:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:51 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:37:51.906 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:51.949 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:51.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:52.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:52.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:52.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:52.148 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 49 16 76337 76321 97.3147 70.1875 0.0162322 0.0102703 2024-03-24T11:37:52.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:52.227 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:52.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:52.328 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:52.398 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:52.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:52.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:52.529 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:52.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:52.580 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:52.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:52.657 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:52.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:52.660 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:37:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:37:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:37:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:37:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:37:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:37:52.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:37:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:37:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:37:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:37:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:37:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:37:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:52 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:37:52.759 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:52.805 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:52.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:52 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:37:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:52 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:37:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:52 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:37:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:52 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:37:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:52 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:37:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:52 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:37:52.906 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:52.960 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:52.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:53.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:53.108 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:53.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:53.148 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 50 15 77322 77307 96.601 61.625 0.0165643 0.0103472 2024-03-24T11:37:53.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:53.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:53.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:53.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:53.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:53.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:53.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:37:53.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:53.497 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T11:37:53.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:53.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:53.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:53.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:53.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: pgmap v875: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 932 op/s 2024-03-24T11:37:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:37:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:37:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:37:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:37:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:37:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/403635857' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: pgmap v875: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 932 op/s 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:37:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:37:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/403635857' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:37:53.765 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:53.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:53.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:53.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: pgmap v875: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 932 op/s 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:37:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/403635857' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:37:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:53.949 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:53.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:54.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:54.079 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:54.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:54.148 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 51 16 78395 78379 96.0206 67 0.0140863 0.0104094 2024-03-24T11:37:54.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:54.230 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:54.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:54.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:54.382 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:54.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:54.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:37:54.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:54.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:54.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:54.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:54.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:54.652 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:54.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:54.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:54.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:54.786 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:54.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:54.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/403635857' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: osdmap e406: 8 total, 8 up, 7 in 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:37:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:54 smithi181 ceph-mon[33998]: osdmap e407: 8 total, 8 up, 7 in 2024-03-24T11:37:54.917 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:54.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:37:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:37:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/403635857' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:37:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: osdmap e406: 8 total, 8 up, 7 in 2024-03-24T11:37:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:37:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:37:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[29608]: osdmap e407: 8 total, 8 up, 7 in 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/403635857' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: osdmap e406: 8 total, 8 up, 7 in 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:37:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:54 smithi179 ceph-mon[37907]: osdmap e407: 8 total, 8 up, 7 in 2024-03-24T11:37:55.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:55.048 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:55.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:55.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:55.149 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 52 16 79523 79507 95.5298 70.5 0.0161325 0.0104632 2024-03-24T11:37:55.179 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:55.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:55.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:55.310 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:55.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:55.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:55.450 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:55.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:55.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:37:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:37:55.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:55.594 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:55.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:55.695 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:55.761 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:55.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:55.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:37:55.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:37:55.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:55.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[29608]: pgmap v878: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:37:55.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:37:55.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:37:55.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:37:55.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:37:55.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:55.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[37907]: pgmap v878: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:37:55.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:37:55.858 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:55 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:37:55.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:55.890 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:55.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:55.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:55 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:37:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:55 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:37:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:37:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:55 smithi181 ceph-mon[33998]: pgmap v878: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T11:37:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:55 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:37:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:55 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:37:55.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:56.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:56.123 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:56.148 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 53 16 80668 80652 95.0776 71.5625 0.00571874 0.0105115 2024-03-24T11:37:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.163 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:56.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:56.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:56.292 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:56.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:56.442 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:56.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:56.582 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:56.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:56.712 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:56.779 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:37:56.779 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:37:56.779 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:37:56.779 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:37:56.780 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:37:56.780 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:37:56.780 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:37:56.780 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:56 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:37:56.813 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:56.853 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:56 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:37:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:56 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:37:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:56 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:37:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:56 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:37:56.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:56.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:56.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:57.094 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:57.139 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:57.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:57.148 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 54 16 81743 81727 94.5611 67.1875 0.0154327 0.0105702 2024-03-24T11:37:57.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:57.277 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:37:57.290 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:57.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:57.330 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:37:57.375 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:37:57.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:57.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:57.424 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:37:57.464 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:37:57.485 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:37:57.486 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:37:57.514 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:37:57.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:57.544 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:57.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:57.556 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:37:57.593 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:37:57.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[29608]: pgmap v879: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 719 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-24T11:37:57.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:37:57.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:37:57.594 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/745442341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:57.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[37907]: pgmap v879: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 719 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-24T11:37:57.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:37:57.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:37:57.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/745442341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:37:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:37:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:37:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:37:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:37:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:37:57.602 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:37:57.603 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:37:57.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:57.687 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:57.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:57.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:57.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:57.815 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:57.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:57 smithi181 ceph-mon[33998]: pgmap v879: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 719 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-24T11:37:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:57 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:37:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:57 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:37:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/745442341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:37:57.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:57.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:57.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:58.046 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:58.048 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T11:37:58.141 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:58.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:58.148 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 55 16 82773 82757 94.0123 64.375 0.0164506 0.0106316 2024-03-24T11:37:58.243 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:58.304 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:58.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:58.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:58.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:58.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:58.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:37:58.550 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:58 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:37:58.551 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:58 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:37:58.588 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:58.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:37:58.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:58.716 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:58.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:58.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:37:58.847 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:58.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:58 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:37:58.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:58 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:37:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:58.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:37:58.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T11:37:58.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:58 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:37:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:58 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:37:58.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:37:58.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:58.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:37:59.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:37:59.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:59.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:37:59.148 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 56 16 83841 83825 93.5255 66.75 0.00455217 0.0106869 2024-03-24T11:37:59.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:37:59.253 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:59.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:37:59.355 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:59.396 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:59.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:59.498 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:37:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:59.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:59.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:37:59.636 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:37:59.668 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:37:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:59.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:37:59.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:59.817 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:59.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:59 smithi181 ceph-mon[33998]: pgmap v880: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 860 op/s 2024-03-24T11:37:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:59 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:37:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:37:59 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:37:59.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:37:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:37:59.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:37:59.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:37:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:59 smithi179 ceph-mon[29608]: pgmap v880: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 860 op/s 2024-03-24T11:37:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:59 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:37:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:37:59 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:37:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:59 smithi179 ceph-mon[37907]: pgmap v880: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 860 op/s 2024-03-24T11:37:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:59 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:37:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:37:59 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:38:00.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:00.086 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:00.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:00.151 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 57 16 85006 84990 93.1583 72.8125 0.0160072 0.0107297 2024-03-24T11:38:00.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:00.219 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:00.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:00.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:00.348 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:00.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:00.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:00.478 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:00.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:00.580 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:00.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:00.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:00.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:00.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:00.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:00.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:00.886 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:00.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:00 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:38:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:00 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:38:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:00 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:38:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:00 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:38:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:00 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:38:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:00 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:38:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:00 smithi181 ceph-mon[33998]: osdmap e408: 8 total, 8 up, 7 in 2024-03-24T11:38:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:38:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:38:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:38:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[29608]: osdmap e408: 8 total, 8 up, 7 in 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:38:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:00 smithi179 ceph-mon[37907]: osdmap e408: 8 total, 8 up, 7 in 2024-03-24T11:38:00.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:01.028 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:01.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:01.032 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:01.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:01.154 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 58 16 86122 86106 92.7512 69.75 0.00793916 0.0107772 2024-03-24T11:38:01.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:01.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:01.247 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:01.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:01.323 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:01.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:01.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:01.471 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:01.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:01.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:01.614 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:01.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:01.715 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:01.750 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:01.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:01.845 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[29608]: pgmap v882: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 737 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[37907]: pgmap v882: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 737 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:38:01.846 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:01 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:38:01.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:01.878 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:01.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:01 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:38:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:01 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:38:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:01 smithi181 ceph-mon[33998]: pgmap v882: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 737 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T11:38:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:01 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:38:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:01 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:38:01.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.027 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:02.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:02.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:02.155 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 59 16 87252 87236 92.3736 70.625 0.0159094 0.0108196 2024-03-24T11:38:02.179 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:02.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:02.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:02.310 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:02.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:02.439 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:02.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:02.549 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:02.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:02.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:02.712 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:02.814 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:02.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[29608]: 3.18 scrub starts 2024-03-24T11:38:02.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[29608]: 3.18 scrub ok 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[29608]: 6.e scrub starts 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[29608]: 6.e scrub ok 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[29608]: 3.3 deep-scrub starts 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[29608]: 3.3 deep-scrub ok 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[37907]: 3.18 scrub starts 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[37907]: 3.18 scrub ok 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[37907]: 6.e scrub starts 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[37907]: 6.e scrub ok 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[37907]: 3.3 deep-scrub starts 2024-03-24T11:38:02.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:02 smithi179 ceph-mon[37907]: 3.3 deep-scrub ok 2024-03-24T11:38:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.862 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:02.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:02.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:02 smithi181 ceph-mon[33998]: 3.18 scrub starts 2024-03-24T11:38:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:02 smithi181 ceph-mon[33998]: 3.18 scrub ok 2024-03-24T11:38:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:02 smithi181 ceph-mon[33998]: 6.e scrub starts 2024-03-24T11:38:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:02 smithi181 ceph-mon[33998]: 6.e scrub ok 2024-03-24T11:38:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:02 smithi181 ceph-mon[33998]: 3.3 deep-scrub starts 2024-03-24T11:38:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:02 smithi181 ceph-mon[33998]: 3.3 deep-scrub ok 2024-03-24T11:38:02.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:02.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:02.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:03.076 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:38:03.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:03.156 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:38:03.156554+0000 min lat: 0.00109547 max lat: 0.156982 avg lat: 0.0108606 2024-03-24T11:38:03.156 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:38:03.156 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 60 16 88395 88379 92.0238 71.4375 0.0126235 0.0108606 2024-03-24T11:38:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:03.164 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:03.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:03.265 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:03.324 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:03.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:03.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:03.456 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:03.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:03.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:03.601 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:03.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:03.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:03 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:38:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:03 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:38:03.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:03 smithi179 ceph-mon[29608]: pgmap v883: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 631 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-24T11:38:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:03 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:38:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:03 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:38:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:03 smithi179 ceph-mon[37907]: pgmap v883: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 631 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-24T11:38:03.742 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:03.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:03.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:03.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:03.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:03.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:03 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:38:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:03 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:38:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:03 smithi181 ceph-mon[33998]: pgmap v883: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 631 B/s rd, 55 MiB/s wr, 887 op/s 2024-03-24T11:38:03.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:04.008 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:04.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.138 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:04.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:04.156 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 61 16 89554 89538 91.7027 72.4375 0.00841874 0.0108995 2024-03-24T11:38:04.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:04.276 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:04.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:04.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:04.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.563 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:04.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:04.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:38:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:38:04.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:04.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:04.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:04.830 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:04 smithi181 ceph-mon[33998]: 1.8 scrub starts 2024-03-24T11:38:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:04 smithi181 ceph-mon[33998]: 1.8 scrub ok 2024-03-24T11:38:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:04 smithi181 ceph-mon[33998]: osdmap e409: 8 total, 8 up, 7 in 2024-03-24T11:38:04.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[29608]: 1.8 scrub starts 2024-03-24T11:38:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[29608]: 1.8 scrub ok 2024-03-24T11:38:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[29608]: osdmap e409: 8 total, 8 up, 7 in 2024-03-24T11:38:04.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[37907]: 1.8 scrub starts 2024-03-24T11:38:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[37907]: 1.8 scrub ok 2024-03-24T11:38:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:04 smithi179 ceph-mon[37907]: osdmap e409: 8 total, 8 up, 7 in 2024-03-24T11:38:04.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:04.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:04.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:05.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:05.103 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:05.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:05.156 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 62 16 90670 90654 91.3486 69.75 0.0111751 0.0109421 2024-03-24T11:38:05.205 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:05.232 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:05.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:05.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:05.372 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:05.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:05.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:38:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:38:05.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:05.497 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:05.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:05.599 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:05.629 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:05.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:05.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:05.760 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:05.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:05.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:05.893 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:05.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: pgmap v885: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:38:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:05 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:38:05.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: pgmap v885: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:38:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: pgmap v885: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:38:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:38:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:38:05.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:05 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:38:05.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.019 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:06.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:06.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:06.149 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:06.156 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 63 16 91621 91605 90.8422 59.4375 0.0063457 0.0110029 2024-03-24T11:38:06.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:06.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:06.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:06.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:06.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:06.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:06.657 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:06.680 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:06.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:06.813 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:06 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:38:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:06 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:38:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:06 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:38:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:06 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:38:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:06 smithi181 ceph-mon[33998]: 6.4 scrub starts 2024-03-24T11:38:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:06 smithi181 ceph-mon[33998]: 6.4 scrub ok 2024-03-24T11:38:06.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:06.954 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:06.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:38:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:38:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:38:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:38:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[29608]: 6.4 scrub starts 2024-03-24T11:38:06.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[29608]: 6.4 scrub ok 2024-03-24T11:38:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:38:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:38:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:38:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:38:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[37907]: 6.4 scrub starts 2024-03-24T11:38:06.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:06 smithi179 ceph-mon[37907]: 6.4 scrub ok 2024-03-24T11:38:07.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:07.083 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:07.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:07.156 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 64 16 92653 92637 90.4307 64.5 0.0163248 0.0110537 2024-03-24T11:38:07.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:07.216 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:07.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:07.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:07.358 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:07.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:07.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:07.491 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:07.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:07.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:07.630 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:07.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:07.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:07.771 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:07.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:07.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:07 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:38:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:07 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:38:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:07 smithi181 ceph-mon[33998]: pgmap v886: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-24T11:38:07.904 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:07.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:07.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:07 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:38:07.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:07 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:38:07.954 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:07 smithi179 ceph-mon[29608]: pgmap v886: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-24T11:38:07.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:07 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:38:07.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:07 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:38:07.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:07 smithi179 ceph-mon[37907]: pgmap v886: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-24T11:38:08.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:08.049 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:08.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:08.152 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:08.157 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 65 16 93781 93765 90.1234 70.5 0.0117635 0.011091 2024-03-24T11:38:08.193 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:08.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:08.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:08.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:08.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:08.425 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:08.466 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:08.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:08.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:08.600 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:08.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:08.701 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:08.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:08.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:08.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:08.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:08.865 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:08.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:08 smithi181 ceph-mon[33998]: 3.1f scrub starts 2024-03-24T11:38:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:08 smithi181 ceph-mon[33998]: 3.1f scrub ok 2024-03-24T11:38:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:08 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:08 smithi179 ceph-mon[29608]: 3.1f scrub starts 2024-03-24T11:38:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:08 smithi179 ceph-mon[29608]: 3.1f scrub ok 2024-03-24T11:38:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:08 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:08 smithi179 ceph-mon[37907]: 3.1f scrub starts 2024-03-24T11:38:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:08 smithi179 ceph-mon[37907]: 3.1f scrub ok 2024-03-24T11:38:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:08 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:38:08.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:09.021 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:09.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:09.151 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:09.157 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 66 16 94662 94646 89.5922 55.0625 0.0251923 0.0111564 2024-03-24T11:38:09.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:09.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:09.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:09.417 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:09.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:09.549 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:09.651 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:09.682 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:09.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:09.812 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:09.882 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:09 smithi179 ceph-mon[29608]: pgmap v887: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:38:09.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:09 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:38:09.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:09 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:38:09.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:09 smithi179 ceph-mon[37907]: pgmap v887: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:38:09.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:09 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:38:09.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:09 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:38:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:09 smithi181 ceph-mon[33998]: pgmap v887: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:38:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:09 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:38:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:09 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:38:09.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:09.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:09.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:10.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:10.096 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:10.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:10.157 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 67 16 95719 95703 89.2409 66.0625 0.00408595 0.0112001 2024-03-24T11:38:10.197 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:10.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:10.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:10.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:10.394 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:10.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:10.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:10.524 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:10.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:10.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:10.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:10.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:38:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:38:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:38:10.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:10.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:10 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:38:10.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:10.782 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:38:10.782 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:38:10.806 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:10.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:10 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:38:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:10 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:38:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:10 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:38:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:10 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:38:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:10 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:38:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:10 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:38:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:10 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:38:10.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:10.938 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:10.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:11.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:11.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:11.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:11.157 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 68 16 96739 96723 88.866 63.75 0.0279271 0.0112471 2024-03-24T11:38:11.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:11.196 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:11.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:11.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:11.332 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:11.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:11.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:11.459 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:11.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:11.561 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:11.590 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:11.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:11.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:11.732 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:11.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:11.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T11:38:11.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:11.878 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:11.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:11.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: pgmap v888: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:38:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:11 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/442517695' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:38:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:38:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:38:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: pgmap v888: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:38:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:38:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/442517695' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: pgmap v888: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:38:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:38:11.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:38:11.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:11 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/442517695' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:38:12.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:12.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:12.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:12.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:12.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:12.158 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 69 16 97802 97786 88.5405 66.4375 0.0118729 0.0112894 2024-03-24T11:38:12.165 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:12.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:12.245 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:38:12.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:12.316 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:12.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:12.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:12.460 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:12.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:12.562 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:12.613 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:12.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:12.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:12.754 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:12.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:12.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:12.888 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:12.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:12.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:12 smithi181 ceph-mon[33998]: 6.1d deep-scrub starts 2024-03-24T11:38:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:12 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:12 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/442517695' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:38:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:12 smithi181 ceph-mon[33998]: osdmap e410: 8 total, 8 up, 7 in 2024-03-24T11:38:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[29608]: 6.1d deep-scrub starts 2024-03-24T11:38:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/442517695' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:38:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[29608]: osdmap e410: 8 total, 8 up, 7 in 2024-03-24T11:38:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[37907]: 6.1d deep-scrub starts 2024-03-24T11:38:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/442517695' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:38:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:12 smithi179 ceph-mon[37907]: osdmap e410: 8 total, 8 up, 7 in 2024-03-24T11:38:12.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:13.027 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:13.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:13.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:13.158 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 70 16 98857 98841 88.2175 65.9375 0.0194237 0.0113301 2024-03-24T11:38:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:13.196 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:13.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:13.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:13.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:13.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:13.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:13.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:13.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:13.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:13.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:13.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:13.637 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T11:38:13.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:38:13.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:38:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: 6.1d deep-scrub ok 2024-03-24T11:38:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: 6.8 deep-scrub starts 2024-03-24T11:38:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: 6.8 deep-scrub ok 2024-03-24T11:38:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: pgmap v890: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 631 B/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:38:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4104415111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:38:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: 6.1d deep-scrub ok 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: 6.8 deep-scrub starts 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: 6.8 deep-scrub ok 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: pgmap v890: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 631 B/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4104415111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:38:13.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:13 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:38:13.731 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:13.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:13.835 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:13.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:13.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:13.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: 6.1d deep-scrub ok 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: 6.8 deep-scrub starts 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: 6.8 deep-scrub ok 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: pgmap v890: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 631 B/s rd, 54 MiB/s wr, 866 op/s 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4104415111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:38:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:13 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:38:13.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.007 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:14.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:14.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:14.151 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:14.158 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 71 16 99940 99924 87.9284 67.6875 0.0166902 0.0113681 2024-03-24T11:38:14.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:14.283 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:14.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:14.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:14.514 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:14.558 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:14.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:38:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:38:14.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:14.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:14.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:14.820 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:14 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:38:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:14 smithi181 ceph-mon[33998]: osdmap e411: 8 total, 8 up, 7 in 2024-03-24T11:38:14.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:14.945 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:14 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:38:14.945 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:14 smithi179 ceph-mon[29608]: osdmap e411: 8 total, 8 up, 7 in 2024-03-24T11:38:14.946 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:14 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:38:14.946 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:14 smithi179 ceph-mon[37907]: osdmap e411: 8 total, 8 up, 7 in 2024-03-24T11:38:14.952 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:14.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:15.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:15.081 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:15.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:15.159 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 72 16 101112 101096 87.7225 73.25 0.0166544 0.0113946 2024-03-24T11:38:15.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:38:15.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:15.187 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:15.210 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:15.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:15.211 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:38:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:38:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:38:15.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:15.370 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:15.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:15.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:15.518 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:15.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:15.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:15.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:15.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:15.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:15.809 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:15.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:15.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T11:38:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:15 smithi181 ceph-mon[33998]: pgmap v892: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:38:15.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:15.930 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:15.930 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:15 smithi179 ceph-mon[29608]: pgmap v892: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:38:15.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:15.931 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:15 smithi179 ceph-mon[37907]: pgmap v892: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:38:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:15.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:15.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:16.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:16.093 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:16.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:16.160 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 73 16 102162 102146 87.4197 65.625 0.0185385 0.0114337 2024-03-24T11:38:16.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:16.255 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:16.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:16.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:16.409 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:16.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:16.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:16.545 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:16.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:16.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:16.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:16.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:16.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:16.824 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:16.825 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:16.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:16.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:16.951 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:16.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:17.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:17.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:17.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:17.160 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 74 16 103257 103241 87.1632 68.4375 0.00788011 0.0114685 2024-03-24T11:38:17.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:17.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:17.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:17.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:17.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:17.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:17.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:17.471 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:17.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:17.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:17.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:17.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:17.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:17.751 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:17.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:17.761 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:38:17.805 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:17 smithi179 ceph-mon[29608]: pgmap v893: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-24T11:38:17.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:17 smithi179 ceph-mon[37907]: pgmap v893: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-24T11:38:17.809 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:38:17.852 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:38:17.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:17.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:17.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:17.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:17 smithi181 ceph-mon[33998]: pgmap v893: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-24T11:38:17.905 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:38:17.945 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:38:17.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:17.991 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:38:18.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:18.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:18.030 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:38:18.068 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:38:18.074 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:38:18.074 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:38:18.074 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:38:18.074 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:38:18.074 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:38:18.075 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:38:18.075 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:38:18.075 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:38:18.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:18.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 75 16 104313 104297 86.8786 66 0.0170822 0.0115052 2024-03-24T11:38:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:18.166 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:18.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:18.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:18.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:18.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:18.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:18.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:18.463 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:18.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:18.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:18.599 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:18.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:18.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:18.740 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:18.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:18.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:18.871 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:18.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:18.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:18 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4233242434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:18.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:18 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4233242434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:18.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:19.002 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:19.104 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:19.146 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:19.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:18 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4233242434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:19.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 76 16 105465 105449 86.6828 72 0.029311 0.0115313 2024-03-24T11:38:19.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:19.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:19.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.412 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:19.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:19.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:19.543 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:19.645 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:19.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:19.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:19.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:19.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:19.952 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:19.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:19 smithi179 ceph-mon[37907]: pgmap v894: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 825 op/s 2024-03-24T11:38:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:38:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:19 smithi179 ceph-mon[29608]: pgmap v894: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 825 op/s 2024-03-24T11:38:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:38:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:19.994 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:20.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:20.081 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:20.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:19 smithi181 ceph-mon[33998]: pgmap v894: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 825 op/s 2024-03-24T11:38:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:38:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:20.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 77 16 106649 106633 86.5181 74 0.0115265 0.0115539 2024-03-24T11:38:20.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:20.228 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:20.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:20.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:20.358 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:20.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:20.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:20.492 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:20.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:20.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:20.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:20.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:20.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:20.754 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:20.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:20.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:20.886 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:20.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:20.987 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:21.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:21.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:21.050 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:38:21.050 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:38:21.050 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:38:21.050 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:38:21.050 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:38:21.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:21.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 78 16 107743 107727 86.2855 68.375 0.00895568 0.0115847 2024-03-24T11:38:21.172 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:21.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:21.274 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:21.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:21.318 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:21.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:21.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:21.452 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:21.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:21.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:21.594 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:21.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:21.696 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:21.745 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:21.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:21.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:21.760 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:38:21.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:21.911 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:21.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:22.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:22.059 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T11:38:22.060 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:21 smithi179 ceph-mon[29608]: pgmap v895: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.30k op/s 2024-03-24T11:38:22.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:21 smithi179 ceph-mon[37907]: pgmap v895: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.30k op/s 2024-03-24T11:38:22.067 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:22.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:21 smithi181 ceph-mon[33998]: pgmap v895: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.30k op/s 2024-03-24T11:38:22.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 79 16 108850 108834 86.0689 69.1875 0.00376442 0.0116135 2024-03-24T11:38:22.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:22.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:22.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:22.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:22.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:22.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:22.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:22.503 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:22.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:22.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:22.636 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:22.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:22.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:22.767 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:22.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:22.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:22.897 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:22.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:22.999 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.027 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:23.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:23.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.154 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:23.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:23.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:2024-03-24T11:38:23.163262+0000 min lat: 0.00109547 max lat: 0.173578 avg lat: 0.0116494 2024-03-24T11:38:23.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T11:38:23.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 80 16 109846 109830 85.7712 62.25 0.00672631 0.0116494 2024-03-24T11:38:23.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.286 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:23.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:23.387 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:23.429 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:23.531 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:23.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:23.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:23.700 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:23.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:23.852 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:23.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:23.979 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:23.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:24.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:24.107 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:24.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:24.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:23 smithi181 ceph-mon[33998]: pgmap v896: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:38:24.162 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 81 16 110796 110780 85.4453 59.375 0.0104213 0.0116982 2024-03-24T11:38:24.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:23 smithi179 ceph-mon[29608]: pgmap v896: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:38:24.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:23 smithi179 ceph-mon[37907]: pgmap v896: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-24T11:38:24.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:24.233 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:24.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:24.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:24.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:24.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:24.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:24.492 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:24.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:24.593 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:24.638 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:24.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:24.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:38:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:38:24.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:24.769 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:24.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:24.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:24.910 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:24.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:25.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:25.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:25.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:25.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:25.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 82 16 111727 111711 85.1129 58.1875 0.0106271 0.0117446 2024-03-24T11:38:25.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:25.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:25.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:25.326 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:25.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:25.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:25.457 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:25.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:25.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:38:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:38:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:38:25.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:25.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:25.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:25.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:25.740 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:25.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:25.843 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:25.874 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:25.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:25.898 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:25 smithi179 ceph-mon[29608]: pgmap v897: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 946 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:38:25.918 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:25.944 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:25.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:26.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:26.088 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:26.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:25 smithi181 ceph-mon[33998]: pgmap v897: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 946 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:38:26.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 83 16 112750 112734 84.8578 63.9375 0.0159845 0.0117794 2024-03-24T11:38:26.191 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:26.218 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:26.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:25 smithi179 ceph-mon[37907]: pgmap v897: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 946 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T11:38:26.234 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:26.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:26.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:26.377 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:26.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:26.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:26.515 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:26.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:26.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:26.647 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:26.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:26.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:26.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:26.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:26.880 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:26.920 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:26.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:26.921 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:38:27.021 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:27.064 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:27.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:27.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:26 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2538002989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:38:27.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 84 16 113828 113812 84.6497 67.375 0.0222599 0.0118083 2024-03-24T11:38:27.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:27.169 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:26 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2538002989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:38:27.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:26 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2538002989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:38:27.192 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:27.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:27.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:27.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:27.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:27.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:27.455 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:27.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:27.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:27.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:27.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:27.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:27.727 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:27.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:27.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:27.871 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:27.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:27.923 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T11:38:27.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:27.999 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:28.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:28.129 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:28.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 85 16 114875 114859 84.4236 65.4375 0.00931873 0.0118402 2024-03-24T11:38:28.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[29608]: pgmap v898: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:38:28.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:28.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2538002989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[29608]: osdmap e412: 8 total, 8 up, 7 in 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1225834367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[37907]: pgmap v898: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2538002989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[37907]: osdmap e412: 8 total, 8 up, 7 in 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1225834367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:38:28.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:27 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:38:28.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.268 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:28.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:28.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:28.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:27 smithi181 ceph-mon[33998]: pgmap v898: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T11:38:28.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:27 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:28.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:27 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2538002989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:38:28.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:27 smithi181 ceph-mon[33998]: osdmap e412: 8 total, 8 up, 7 in 2024-03-24T11:38:28.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:27 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1225834367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:38:28.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:27 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:38:28.409 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:28.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:28.541 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:28.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.677 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:28.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:28.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:28.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:28.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:28.963 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:28.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:29.067 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:29.111 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:29.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:29.162 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:28 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:38:29.162 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:28 smithi179 ceph-mon[29608]: osdmap e413: 8 total, 8 up, 7 in 2024-03-24T11:38:29.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:28 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:38:29.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:28 smithi179 ceph-mon[37907]: osdmap e413: 8 total, 8 up, 7 in 2024-03-24T11:38:29.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 86 16 115991 115975 84.253 69.75 0.0103322 0.0118642 2024-03-24T11:38:29.214 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:29.253 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:29.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:29.355 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:29.398 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:29.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:29.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:28 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:38:29.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:28 smithi181 ceph-mon[33998]: osdmap e413: 8 total, 8 up, 7 in 2024-03-24T11:38:29.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:29.537 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:29.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:29.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:29.666 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:29.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:29.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:29.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:29.808 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:29.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:29.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:29.944 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:29.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:30.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-24T11:38:30.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:30.118 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:30.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:30.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 87 16 117092 117076 84.0755 68.8125 0.0178612 0.0118887 2024-03-24T11:38:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[29608]: pgmap v901: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 825 op/s 2024-03-24T11:38:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:38:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:38:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[29608]: osdmap e414: 8 total, 8 up, 7 in 2024-03-24T11:38:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[29608]: osdmap e415: 8 total, 8 up, 7 in 2024-03-24T11:38:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[37907]: pgmap v901: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 825 op/s 2024-03-24T11:38:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:38:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:38:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[37907]: osdmap e414: 8 total, 8 up, 7 in 2024-03-24T11:38:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:29 smithi179 ceph-mon[37907]: osdmap e415: 8 total, 8 up, 7 in 2024-03-24T11:38:30.219 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:30.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:30.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:30.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:30.376 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:30.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:30.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:29 smithi181 ceph-mon[33998]: pgmap v901: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.8 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 825 op/s 2024-03-24T11:38:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:29 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:38:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:29 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:38:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:29 smithi181 ceph-mon[33998]: osdmap e414: 8 total, 8 up, 7 in 2024-03-24T11:38:30.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:29 smithi181 ceph-mon[33998]: osdmap e415: 8 total, 8 up, 7 in 2024-03-24T11:38:30.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:30.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:30.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:30.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:30.654 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:30.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:30.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:30.783 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:30.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:30.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:30.927 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:30.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:31.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:31.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:31.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:31.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:31.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 88 16 118214 118198 83.917 70.125 0.00856957 0.0119115 2024-03-24T11:38:31.186 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:31.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:31.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:38:31.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:31.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:31.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:31.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:31.481 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:31.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:31.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:31.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:31.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:31.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:31.743 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:31.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:31.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:31.874 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:31.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:31.976 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:32.015 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:32.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:32.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:32.117 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:32.159 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:38:32.159 INFO:teuthology.orchestra.run.smithi179.stdout: "size": 20, 2024-03-24T11:38:32.159 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 600, 2024-03-24T11:38:32.159 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [ 2024-03-24T11:38:32.159 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:183295 10.14 10:2c9f56c8:::benchmark_data_smithi179_155357_object72799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:35:58.004331+0000", 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 153.742835418, 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.518664601, 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 183295 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.160 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004331+0000", 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004331+0000", 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004333+0000", 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.658e-06 2024-03-24T11:38:32.161 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004338+0000", 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8119999999999998e-06 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004348+0000", 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0261999999999999e-05 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.162 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004354+0000", 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.1700000000000002e-06 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004423+0000", 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9052000000000002e-05 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.163 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.004436+0000", 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2765999999999999e-05 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.513027+0000", 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.50859081900000003 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.513151+0000", 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012417600000000001 2024-03-24T11:38:32.164 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,4", 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.513556+0000", 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00040522399999999998 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.519570+0000", 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0060136149999999999 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.165 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.519581+0000", 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1123e-05 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.522808+0000", 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032270580000000001 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.522933+0000", 2024-03-24T11:38:32.166 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012494900000000001 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:35:58.522996+0000", 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.2916999999999994e-05 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.167 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:190195 10.13 10:ce645201:::benchmark_data_smithi179_155357_object79699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:06.240397+0000", 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 145.506769612, 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.178546131, 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 190195 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.168 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.240397+0000", 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.240397+0000", 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.240398+0000", 2024-03-24T11:38:32.169 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8099999999999998e-07 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.240400+0000", 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.909e-06 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.240407+0000", 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4819999999999999e-06 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.170 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.240412+0000", 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8969999999999999e-06 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.244927+0000", 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045148000000000002 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.245058+0000", 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013091900000000001 2024-03-24T11:38:32.171 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.245507+0000", 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00044951300000000002 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.249568+0000", 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0040611579999999996 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.249580+0000", 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.199e-05 2024-03-24T11:38:32.172 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.418766+0000", 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16918591899999999 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.418874+0000", 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00010743000000000001 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:06.418943+0000", 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9733000000000005e-05 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:213986 10.22 10:46d26a32:::benchmark_data_smithi179_155357_object103490:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:25.292115+0000", 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 126.45505156, 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1226197179999999, 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 213986 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.292115+0000", 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.175 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.292115+0000", 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.292117+0000", 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.928e-06 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.292123+0000", 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5180000000000004e-06 2024-03-24T11:38:32.176 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.292134+0000", 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1472e-05 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.292142+0000", 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.4399999999999999e-06 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.177 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.442174+0000", 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.150032205 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:25.442185+0000", 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1499e-05 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:26.414654+0000", 2024-03-24T11:38:32.178 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97246872399999995 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:26.414735+0000", 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0932000000000004e-05 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:215439 10.1f 10.75d0325f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 124.422101873, 2024-03-24T11:38:32.179 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.99205663799999999, 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 215439 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.180 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.325066+0000", 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.282e-06 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.325072+0000", 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8880000000000002e-06 2024-03-24T11:38:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.325084+0000", 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2232000000000001e-05 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.325092+0000", 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0779999999999996e-06 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.182 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.330945+0000", 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0058526020000000002 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.330953+0000", 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.5820000000000004e-06 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317104+0000", 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.98615179799999997 2024-03-24T11:38:32.183 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317122+0000", 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7176e-05 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:215473 10.1f 10.7723ef1f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:27.349578+0000", 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 124.397588379, 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96758165100000004, 2024-03-24T11:38:32.184 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 215473 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349578+0000", 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.185 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349578+0000", 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349580+0000", 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.468e-06 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.186 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349585+0000", 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2700000000000004e-06 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349599+0000", 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.345e-05 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349607+0000", 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7620000000000001e-06 2024-03-24T11:38:32.187 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349697+0000", 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9394000000000005e-05 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.349707+0000", 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.031e-05 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317152+0000", 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96744464900000005 2024-03-24T11:38:32.188 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317160+0000", 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3480000000000004e-06 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:215489 10.1f 10.ac49479f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:27.360929+0000", 2024-03-24T11:38:32.189 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 124.386237878, 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95626312099999999, 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 215489 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.190 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.360929+0000", 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.360929+0000", 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.360930+0000", 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6199999999999997e-07 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.191 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.360934+0000", 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.1579999999999998e-06 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.360944+0000", 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.73e-06 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.192 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.360951+0000", 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.0140000000000001e-06 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.361003+0000", 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2392999999999998e-05 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.361012+0000", 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.5939999999999994e-06 2024-03-24T11:38:32.193 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317185+0000", 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95617313199999998 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317192+0000", 2024-03-24T11:38:32.194 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.3379999999999999e-06 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:215498 10.1f 10.f192955f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:27.366723+0000", 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 124.380443993, 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95050164800000003, 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.195 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 215498 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366723+0000", 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366723+0000", 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.196 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366724+0000", 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.314e-06 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366729+0000", 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.8149999999999996e-06 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366744+0000", 2024-03-24T11:38:32.197 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4748e-05 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366752+0000", 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.2719999999999997e-06 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366810+0000", 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8077e-05 2024-03-24T11:38:32.198 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.366818+0000", 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4020000000000002e-06 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317216+0000", 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95039784999999999 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317224+0000", 2024-03-24T11:38:32.199 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1699999999999997e-06 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:215505 10.1f 10.e82ab2df (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:27.370347+0000", 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 124.37681963599999, 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94690793699999998, 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.200 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 215505 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370347+0000", 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370347+0000", 2024-03-24T11:38:32.201 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370348+0000", 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4300000000000002e-07 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.202 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370352+0000", 2024-03-24T11:38:32.203 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8369999999999999e-06 2024-03-24T11:38:32.203 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.203 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.203 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.203 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370363+0000", 2024-03-24T11:38:32.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1331999999999999e-05 2024-03-24T11:38:32.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.204 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.204 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.204 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370369+0000", 2024-03-24T11:38:32.204 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7509999999999997e-06 2024-03-24T11:38:32.204 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.205 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.205 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370426+0000", 2024-03-24T11:38:32.205 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7074999999999998e-05 2024-03-24T11:38:32.205 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.205 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.206 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:27.370435+0000", 2024-03-24T11:38:32.206 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6200000000000005e-06 2024-03-24T11:38:32.206 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.206 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.206 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317247+0000", 2024-03-24T11:38:32.207 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94681211499999995 2024-03-24T11:38:32.207 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.207 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.207 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.207 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.317255+0000", 2024-03-24T11:38:32.208 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3639999999999999e-06 2024-03-24T11:38:32.208 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.208 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.208 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.208 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.208 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.208 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:215441 10.22 10:44579354:::benchmark_data_smithi179_155357_object104945:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:38:32.209 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:28.318481+0000", 2024-03-24T11:38:32.209 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 123.42868611, 2024-03-24T11:38:32.209 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1167338490000001, 2024-03-24T11:38:32.209 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.209 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.210 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.210 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 215441 2024-03-24T11:38:32.210 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.210 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.211 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.211 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318481+0000", 2024-03-24T11:38:32.211 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.211 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.211 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318481+0000", 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318482+0000", 2024-03-24T11:38:32.212 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.026e-06 2024-03-24T11:38:32.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.213 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.213 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318487+0000", 2024-03-24T11:38:32.213 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4859999999999997e-06 2024-03-24T11:38:32.213 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.213 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.214 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.214 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318498+0000", 2024-03-24T11:38:32.214 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.102e-05 2024-03-24T11:38:32.214 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.214 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318507+0000", 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6780000000000006e-06 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318559+0000", 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1968000000000001e-05 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.215 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318568+0000", 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6129999999999993e-06 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.431552+0000", 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1129835809999999 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for missing object", 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.432012+0000", 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00045969099999999997 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.216 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.435130+0000", 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0031180230000000001 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.435215+0000", 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4763000000000001e-05 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.217 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:215442 10.22 10:47acaaa5:::benchmark_data_smithi179_155357_object104946:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:28.318529+0000", 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 123.42863797299999, 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.118590204, 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "reached pg", 2024-03-24T11:38:32.218 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 215442 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318529+0000", 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318529+0000", 2024-03-24T11:38:32.219 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318529+0000", 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.58e-07 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318532+0000", 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.785e-06 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318539+0000", 2024-03-24T11:38:32.220 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.007e-06 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318545+0000", 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5140000000000001e-06 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318597+0000", 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2573e-05 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.221 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:28.318604+0000", 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9700000000000002e-06 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.432330+0000", 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.113725396 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for missing object", 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.432459+0000", 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012964000000000001 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.222 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.437040+0000", 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0045803340000000001 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:29.437119+0000", 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.9526999999999999e-05 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:217825 10.1f 10:f983bdde:::benchmark_data_smithi179_155357_object107329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.223 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.403430+0000", 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.34373722300001, 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97977436699999998, 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 217825 2024-03-24T11:38:32.224 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403430+0000", 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.225 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403430+0000", 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403431+0000", 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.128e-06 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403435+0000", 2024-03-24T11:38:32.226 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6129999999999997e-06 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403446+0000", 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0368e-05 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403463+0000", 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7773000000000001e-05 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.227 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403528+0000", 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4147999999999996e-05 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.403537+0000", 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3060000000000003e-06 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.228 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.377763+0000", 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97422630700000001 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.377905+0000", 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000142175 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.229 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.378098+0000", 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000192516 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.382991+0000", 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0048933650000000002 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383047+0000", 2024-03-24T11:38:32.230 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6192e-05 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383108+0000", 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.0266000000000002e-05 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383163+0000", 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5276999999999999e-05 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.231 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383204+0000", 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0933000000000003e-05 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:217834 10.1f 10:fc7dc2b2:::benchmark_data_smithi179_155357_object107338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.407079+0000", 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.340087415, 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97673252399999999, 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.232 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 217834 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407079+0000", 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407079+0000", 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.233 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407081+0000", 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1009999999999999e-06 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407085+0000", 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7609999999999998e-06 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.234 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407096+0000", 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1188e-05 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407104+0000", 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.6429999999999995e-06 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.235 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407590+0000", 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048595000000000002 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.407598+0000", 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.3210000000000005e-06 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.379660+0000", 2024-03-24T11:38:32.236 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.97206176099999997 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.379773+0000", 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000112571 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.379985+0000", 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00021259499999999999 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.237 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383303+0000", 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033176540000000002 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383589+0000", 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00028606399999999999 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.238 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383680+0000", 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.1100000000000005e-05 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383765+0000", 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.4525999999999997e-05 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383812+0000", 2024-03-24T11:38:32.239 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7289000000000001e-05 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:217848 10.1f 10:f847d7ac:::benchmark_data_smithi179_155357_object107352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.418791+0000", 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.32837585599999, 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96820467700000001, 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.240 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 217848 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418791+0000", 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418791+0000", 2024-03-24T11:38:32.241 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418792+0000", 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4750000000000001e-06 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418798+0000", 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8340000000000004e-06 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.242 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418813+0000", 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4751e-05 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418822+0000", 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.8959999999999992e-06 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418890+0000", 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8281000000000006e-05 2024-03-24T11:38:32.243 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.418900+0000", 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0166000000000001e-05 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.380993+0000", 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96209224100000001 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.381140+0000", 2024-03-24T11:38:32.244 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00014713899999999999 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.381277+0000", 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00013738199999999999 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383637+0000", 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.002359655 2024-03-24T11:38:32.245 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.386867+0000", 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032299770000000002 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.386882+0000", 2024-03-24T11:38:32.246 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.4837e-05 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.386945+0000", 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.3522000000000006e-05 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.386996+0000", 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0521e-05 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.247 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:217864 10.1f 10:ff755c06:::benchmark_data_smithi179_155357_object107368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.432029+0000", 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.315138198, 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.96110996999999998, 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 217864 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.248 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432029+0000", 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432029+0000", 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.249 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432030+0000", 2024-03-24T11:38:32.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3400000000000001e-06 2024-03-24T11:38:32.250 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.250 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.250 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.250 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432035+0000", 2024-03-24T11:38:32.250 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4600000000000002e-06 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432052+0000", 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6116999999999998e-05 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432060+0000", 2024-03-24T11:38:32.251 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6810000000000004e-06 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432127+0000", 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7279000000000004e-05 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.432136+0000", 2024-03-24T11:38:32.252 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.9979999999999992e-06 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.383871+0000", 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95173492500000001 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.384000+0000", 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012814599999999999 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.253 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.384242+0000", 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00024206600000000001 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.389940+0000", 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0056984059999999996 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.389991+0000", 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0748000000000003e-05 2024-03-24T11:38:32.254 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393050+0000", 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.003058856 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393096+0000", 2024-03-24T11:38:32.255 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6493999999999999e-05 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393139+0000", 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.2453999999999997e-05 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.256 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:217876 10.1f 10:fa125f40:::benchmark_data_smithi179_155357_object107380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.439026+0000", 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.308140486, 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95419121100000004, 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 217876 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.257 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439026+0000", 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439026+0000", 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439028+0000", 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5069999999999999e-06 2024-03-24T11:38:32.258 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.259 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.259 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439034+0000", 2024-03-24T11:38:32.259 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.6829999999999996e-06 2024-03-24T11:38:32.259 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.259 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.259 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:31 smithi179 ceph-mon[37907]: pgmap v904: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:38:32.260 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.260 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439052+0000", 2024-03-24T11:38:32.260 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.8902999999999998e-05 2024-03-24T11:38:32.260 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.260 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439062+0000", 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.3610000000000007e-06 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439128+0000", 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.6661000000000004e-05 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.439139+0000", 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0417999999999999e-05 2024-03-24T11:38:32.261 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.385455+0000", 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94631632200000004 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.385563+0000", 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000107784 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.262 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.385747+0000", 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00018429899999999999 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.390034+0000", 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0042864269999999998 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.390080+0000", 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6513e-05 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.263 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393150+0000", 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030697810000000002 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393186+0000", 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.6371999999999997e-05 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393218+0000", 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.1180000000000003e-05 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.264 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:217885 10.1f 10:f89a4e5c:::benchmark_data_smithi179_155357_object107389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.445042+0000", 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.302124873, 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.95153739599999998, 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.265 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 217885 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445042+0000", 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.266 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.267 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445042+0000", 2024-03-24T11:38:32.267 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.267 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.267 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.267 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445044+0000", 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.736e-06 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445049+0000", 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2789999999999998e-06 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.268 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445066+0000", 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.7356000000000001e-05 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445074+0000", 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.0129999999999993e-06 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445139+0000", 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.4219000000000003e-05 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.269 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.445147+0000", 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1219999999999995e-06 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.387041+0000", 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.94189472299999999 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.270 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.387293+0000", 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00025189500000000003 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.387485+0000", 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000191898 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393036+0000", 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.005550681 2024-03-24T11:38:32.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393271+0000", 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000235293 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.396472+0000", 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0032006790000000001 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.396529+0000", 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7454000000000003e-05 2024-03-24T11:38:32.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.396579+0000", 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.0047999999999999e-05 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:217967 10.1f 10:fdaca0b3:::benchmark_data_smithi179_155357_object107471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.504698+0000", 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.24246859500001, 2024-03-24T11:38:32.273 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.89206249500000001, 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 217967 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504698+0000", 2024-03-24T11:38:32.274 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504698+0000", 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504699+0000", 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0729999999999999e-06 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.275 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504704+0000", 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.6840000000000004e-06 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504715+0000", 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.1471000000000001e-05 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.276 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504724+0000", 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.1140000000000006e-06 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504781+0000", 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.7429000000000002e-05 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.504790+0000", 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.7679999999999997e-06 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.277 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.388571+0000", 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.88378172399999999 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.388668+0000", 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6467999999999999e-05 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.278 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.388830+0000", 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00016192100000000001 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393261+0000", 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0044307820000000003 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393276+0000", 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.5112e-05 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.279 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.396634+0000", 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0033578560000000002 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.396720+0000", 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 8.6384000000000005e-05 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.396761+0000", 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.0708999999999999e-05 2024-03-24T11:38:32.280 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.15621.0:218000 10.1f 10:fb7d6eab:::benchmark_data_smithi179_155357_object107504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:36:31.532490+0000", 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 120.214676591, 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.86507273699999998, 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.281 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.15621", 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 218000 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532490+0000", 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.282 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532490+0000", 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532491+0000", 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.2640000000000001e-06 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532497+0000", 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.2020000000000003e-06 2024-03-24T11:38:32.283 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532513+0000", 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.6144999999999999e-05 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532521+0000", 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 7.7449999999999995e-06 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532575+0000", 2024-03-24T11:38:32.284 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.4064999999999998e-05 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for peered", 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:31.532581+0000", 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.8109999999999997e-06 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.390931+0000", 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.85834998799999995 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.285 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.391028+0000", 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.6274999999999996e-05 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.391241+0000", 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000213425 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.393280+0000", 2024-03-24T11:38:32.286 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0020386739999999999 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.396338+0000", 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0030577310000000002 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.397474+0000", 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.001136043 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.287 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.397525+0000", 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.1357999999999999e-05 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:36:32.397563+0000", 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 3.8010999999999999e-05 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.288 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25474.0:110800 11.c 11:3a4828e5:::benchmark_data_smithi179_178435_object110799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:38:24.165617+0000", 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.5815493649999999, 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.17625081000000001, 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25474", 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/287749777", 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 110800 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.289 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.290 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165617+0000", 2024-03-24T11:38:32.290 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.290 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.290 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.290 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165617+0000", 2024-03-24T11:38:32.291 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.291 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.291 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.291 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.291 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165619+0000", 2024-03-24T11:38:32.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.0420000000000001e-06 2024-03-24T11:38:32.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.292 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.292 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.292 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165642+0000", 2024-03-24T11:38:32.292 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 2.27e-05 2024-03-24T11:38:32.292 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.293 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.293 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165652+0000", 2024-03-24T11:38:32.293 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0124e-05 2024-03-24T11:38:32.293 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.293 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.294 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165663+0000", 2024-03-24T11:38:32.294 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.0675e-05 2024-03-24T11:38:32.294 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.294 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.294 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165732+0000", 2024-03-24T11:38:32.295 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.9167999999999999e-05 2024-03-24T11:38:32.295 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.295 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.295 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.295 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.165903+0000", 2024-03-24T11:38:32.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00017036699999999999 2024-03-24T11:38:32.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.296 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.296 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.296 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.166051+0000", 2024-03-24T11:38:32.296 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000148181 2024-03-24T11:38:32.296 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.297 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.297 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.169011+0000", 2024-03-24T11:38:32.297 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00296014 2024-03-24T11:38:32.297 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.297 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.298 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.176499+0000", 2024-03-24T11:38:32.298 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0074884230000000001 2024-03-24T11:38:32.298 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.298 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.298 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.341328+0000", 2024-03-24T11:38:32.299 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16482903800000001 2024-03-24T11:38:32.299 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.299 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.299 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.299 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.341384+0000", 2024-03-24T11:38:32.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.5899999999999997e-05 2024-03-24T11:38:32.300 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.300 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.300 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.300 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.341868+0000", 2024-03-24T11:38:32.300 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00048405200000000001 2024-03-24T11:38:32.301 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.301 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.301 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.301 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.301 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.301 INFO:teuthology.orchestra.run.smithi179.stdout: "description": "osd_op(client.25474.0:110808 11.c 11:37c5d60f:::benchmark_data_smithi179_178435_object110807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-03-24T11:38:32.301 INFO:teuthology.orchestra.run.smithi179.stdout: "initiated_at": "2024-03-24T11:38:24.169994+0000", 2024-03-24T11:38:32.302 INFO:teuthology.orchestra.run.smithi179.stdout: "age": 7.5771732099999998, 2024-03-24T11:38:32.302 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.174739318, 2024-03-24T11:38:32.302 INFO:teuthology.orchestra.run.smithi179.stdout: "type_data": { 2024-03-24T11:38:32.302 INFO:teuthology.orchestra.run.smithi179.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:38:32.302 INFO:teuthology.orchestra.run.smithi179.stdout: "client_info": { 2024-03-24T11:38:32.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client": "client.25474", 2024-03-24T11:38:32.303 INFO:teuthology.orchestra.run.smithi179.stdout: "client_addr": "172.21.15.179:0/287749777", 2024-03-24T11:38:32.303 INFO:teuthology.orchestra.run.smithi179.stdout: "tid": 110808 2024-03-24T11:38:32.303 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.303 INFO:teuthology.orchestra.run.smithi179.stdout: "events": [ 2024-03-24T11:38:32.303 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "initiated", 2024-03-24T11:38:32.304 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.169994+0000", 2024-03-24T11:38:32.304 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.304 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.304 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.304 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "header_read", 2024-03-24T11:38:32.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.169994+0000", 2024-03-24T11:38:32.305 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0 2024-03-24T11:38:32.305 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.305 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.305 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "throttled", 2024-03-24T11:38:32.305 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.169995+0000", 2024-03-24T11:38:32.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.3009999999999999e-06 2024-03-24T11:38:32.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.306 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.306 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "all_read", 2024-03-24T11:38:32.306 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.170015+0000", 2024-03-24T11:38:32.306 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 1.9728000000000002e-05 2024-03-24T11:38:32.306 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.307 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.307 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "dispatched", 2024-03-24T11:38:32.307 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.170024+0000", 2024-03-24T11:38:32.307 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 9.2930000000000006e-06 2024-03-24T11:38:32.307 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.308 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "queued_for_pg", 2024-03-24T11:38:32.308 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.170031+0000", 2024-03-24T11:38:32.308 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 6.7209999999999997e-06 2024-03-24T11:38:32.308 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.308 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "reached_pg", 2024-03-24T11:38:32.309 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.170079+0000", 2024-03-24T11:38:32.309 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.7874999999999998e-05 2024-03-24T11:38:32.309 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.309 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.309 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "started", 2024-03-24T11:38:32.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.170207+0000", 2024-03-24T11:38:32.310 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.00012817199999999999 2024-03-24T11:38:32.310 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.310 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.310 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "waiting for subops from 2,7", 2024-03-24T11:38:32.310 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.170345+0000", 2024-03-24T11:38:32.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.000138288 2024-03-24T11:38:32.311 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.311 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.311 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "op_commit", 2024-03-24T11:38:32.311 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.172925+0000", 2024-03-24T11:38:32.311 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0025795390000000001 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.176540+0000", 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.0036153660000000001 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:38:32.312 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.344629+0000", 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 0.16808894199999999 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "commit_sent", 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.344687+0000", 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 5.8479e-05 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-24T11:38:32.313 INFO:teuthology.orchestra.run.smithi179.stdout: { 2024-03-24T11:38:32.314 INFO:teuthology.orchestra.run.smithi179.stdout: "event": "done", 2024-03-24T11:38:32.314 INFO:teuthology.orchestra.run.smithi179.stdout: "time": "2024-03-24T11:38:24.344733+0000", 2024-03-24T11:38:32.314 INFO:teuthology.orchestra.run.smithi179.stdout: "duration": 4.5614000000000001e-05 2024-03-24T11:38:32.314 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.314 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.314 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.314 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-24T11:38:32.315 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-24T11:38:32.315 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:38:32.315 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:31 smithi179 ceph-mon[29608]: pgmap v904: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:38:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:32.317 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:32.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:32.318 INFO:tasks.radosbench.radosbench.0.smithi179.stdout: 89 16 119219 119203 83.6798 62.8125 0.00807351 0.0119456 2024-03-24T11:38:32.318 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:31 smithi181 ceph-mon[33998]: pgmap v904: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-24T11:38:32.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:32.450 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:32.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:32.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:32.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:32.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:32.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:32.726 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:32.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:32.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:32.870 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:32.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:32.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:33.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:33.105 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.144 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:33.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:33.163 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total time run: 90.0068 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Total writes made: 120295 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Write size: 65536 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Object size: 65536 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Bandwidth (MB/sec): 83.5319 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Bandwidth: 27.4971 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max bandwidth (MB/sec): 183.5 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min bandwidth (MB/sec): 55.0625 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average IOPS: 1336 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev IOPS: 439.953 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max IOPS: 2936 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min IOPS: 881 2024-03-24T11:38:33.164 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Average Latency(s): 0.0119675 2024-03-24T11:38:33.165 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Stddev Latency(s): 0.00855916 2024-03-24T11:38:33.165 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Max latency(s): 0.179392 2024-03-24T11:38:33.165 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Min latency(s): 0.00109547 2024-03-24T11:38:33.170 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T11:38:33.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:33.276 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:33.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:33.417 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:33.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.548 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:33.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:33.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:33.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:33.820 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:33.922 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:33.972 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:33.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:34.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:34.102 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:34.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:34.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:33 smithi181 ceph-mon[33998]: pgmap v905: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 926 B/s rd, 59 MiB/s wr, 948 op/s 2024-03-24T11:38:34.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:34.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:33 smithi179 ceph-mon[29608]: pgmap v905: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 926 B/s rd, 59 MiB/s wr, 948 op/s 2024-03-24T11:38:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:34.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:33 smithi179 ceph-mon[37907]: pgmap v905: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 926 B/s rd, 59 MiB/s wr, 948 op/s 2024-03-24T11:38:34.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:34.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:34.234 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:34.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:34.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:34.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:34.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:34.469 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:38:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:38:34.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:34.512 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:34.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:34.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:34.655 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:34.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:34.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:34.790 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:34.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:34.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:34.930 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:34.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:34.987 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T11:38:35.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:35.064 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:35.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:35.136 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:35.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:35.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:35.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:35.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:38:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:38:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:38:35.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:35.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:35.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:35.388 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:35.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:35.475 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:35.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:35.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:35.597 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:35 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:35.597 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:35 smithi179 ceph-mon[29608]: pgmap v906: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.38k op/s 2024-03-24T11:38:35.597 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:35 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:35.597 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:35 smithi179 ceph-mon[37907]: pgmap v906: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.38k op/s 2024-03-24T11:38:35.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:35.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:35.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:35 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:35 smithi181 ceph-mon[33998]: pgmap v906: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.38k op/s 2024-03-24T11:38:35.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:35.736 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:35.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:35.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:35.873 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:35.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:35.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:36.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:36.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:36.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:36.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:36.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.294 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:36.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:36.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:36.399 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2619972288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:36.399 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2619972288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:36.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:36.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.555 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:36.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:36.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2619972288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:36.656 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:36.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:36.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:36.818 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:36.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:36.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:36.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:37.034 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:38:37.034 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:38:37.034 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:38:37.034 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:38:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:38:37.036 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:38:37.036 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:38:37.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:37.100 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:37.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:37.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:37.232 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:37.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:37.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:37.364 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:37.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:37.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:37.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:37.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:37.595 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:37.633 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:37.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:37 smithi181 ceph-mon[33998]: pgmap v907: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 767 B/s rd, 66 MiB/s wr, 1.19k op/s 2024-03-24T11:38:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:37 smithi179 ceph-mon[29608]: pgmap v907: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 767 B/s rd, 66 MiB/s wr, 1.19k op/s 2024-03-24T11:38:37.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:37 smithi179 ceph-mon[37907]: pgmap v907: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 767 B/s rd, 66 MiB/s wr, 1.19k op/s 2024-03-24T11:38:37.735 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:37.764 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:37.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:37.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:37.890 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:37.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:37.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.029 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:38.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:38.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:38.167 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:38.269 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:38.309 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:38.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:38.439 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:38.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:38.572 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:38.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:38.719 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:38.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:38.851 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:38.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:38.997 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:38.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:39.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:39.131 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:39.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:39.232 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:39.269 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:39.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:39.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:39.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:39.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:39.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:39.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:39.551 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:39.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:39.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:39.680 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:39.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:39.767 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:39 smithi179 ceph-mon[37907]: pgmap v908: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 905 op/s 2024-03-24T11:38:39.768 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:39 smithi179 ceph-mon[29608]: pgmap v908: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 905 op/s 2024-03-24T11:38:39.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:39.818 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:39.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:39.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:39 smithi181 ceph-mon[33998]: pgmap v908: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 905 op/s 2024-03-24T11:38:39.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:39.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:39.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:40.058 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:40.099 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:40.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:40.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:40.238 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:40.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:40.340 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:40.376 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:40.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:40.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:40.508 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:40.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:40.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:40.651 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:40.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:40.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:40 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:38:40.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:40 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:38:40.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:40 smithi179 ceph-mon[29608]: osdmap e416: 8 total, 8 up, 7 in 2024-03-24T11:38:40.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:40 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:38:40.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:40 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:38:40.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:40 smithi179 ceph-mon[37907]: osdmap e416: 8 total, 8 up, 7 in 2024-03-24T11:38:40.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:40.794 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:40.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:40.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:40.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:40 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:38:40.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:40 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:38:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:40 smithi181 ceph-mon[33998]: osdmap e416: 8 total, 8 up, 7 in 2024-03-24T11:38:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:40.935 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:40.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:41.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:41.065 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:41.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:41.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:41.220 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:41.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:41.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:41.359 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:41.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:41.366 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:38:41.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:41.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:41.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:41.588 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:41.655 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:41.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:41.697 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:41 smithi179 ceph-mon[29608]: pgmap v910: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 28 MiB/s wr, 1.12k op/s 2024-03-24T11:38:41.697 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:41 smithi179 ceph-mon[37907]: pgmap v910: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 28 MiB/s wr, 1.12k op/s 2024-03-24T11:38:41.722 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:41.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:41.806 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:41.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:41.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:41 smithi181 ceph-mon[33998]: pgmap v910: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 28 MiB/s wr, 1.12k op/s 2024-03-24T11:38:41.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:41.936 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:41.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:42.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:42.067 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:42.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:42.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:42.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:42.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:42.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:42.338 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:42.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:42.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:42.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:42.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:42.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:42.654 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:42.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:42.756 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:42.799 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:42.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:42.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:42.930 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:42.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:43.032 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:43.067 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:43.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:43.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:43.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:43.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:43.200 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:43.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:43.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:43.342 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:43.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:43.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:43.472 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:43.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:43.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:43.611 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:43.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:43.713 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:43.753 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:43.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:43.855 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:43.897 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:43.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:43.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:43 smithi181 ceph-mon[33998]: pgmap v911: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 28 MiB/s wr, 1.12k op/s 2024-03-24T11:38:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:43 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2970193540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:38:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:43 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:38:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:43 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:38:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:43 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:38:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[29608]: pgmap v911: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 28 MiB/s wr, 1.12k op/s 2024-03-24T11:38:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2970193540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:38:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:38:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:38:43.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:38:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[37907]: pgmap v911: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 28 MiB/s wr, 1.12k op/s 2024-03-24T11:38:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2970193540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:38:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:38:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:38:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:43 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:38:43.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.041 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:44.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:44.142 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:44.170 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:44.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.323 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:44.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:44.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:44.461 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:44.495 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.7 to 1 (10000) 2024-03-24T11:38:44.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:38:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:38:44.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:44.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:44.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:44.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:44.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:44.849 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:44 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:38:44.850 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:44 smithi181 ceph-mon[33998]: osdmap e417: 8 total, 8 up, 7 in 2024-03-24T11:38:44.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:44.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:44 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:38:44.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:44 smithi179 ceph-mon[29608]: osdmap e417: 8 total, 8 up, 7 in 2024-03-24T11:38:44.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:44 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:38:44.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:44 smithi179 ceph-mon[37907]: osdmap e417: 8 total, 8 up, 7 in 2024-03-24T11:38:44.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:44.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:44.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:45.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:45.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:45.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:45.222 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:45.258 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:45.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:45.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:45.376 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:38:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:38:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:38:45.386 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:45.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:45.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:45.537 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:45.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:45.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:45.677 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:45.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:45 smithi179 ceph-mon[29608]: pgmap v913: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:38:45.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:45.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:45 smithi179 ceph-mon[37907]: pgmap v913: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:38:45.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:45.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:45 smithi181 ceph-mon[33998]: pgmap v913: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-24T11:38:45.919 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:45.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:45.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:46.058 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:46.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:46.100 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:46.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:46.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:46.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:46.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:46.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:46.377 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:46.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:46.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:46.516 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:46.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:46.571 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T11:38:46.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:38:46.574 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:38:46.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:46.675 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:46.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:46.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:46.807 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:46.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/726041406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:38:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:46 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:38:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:46 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:38:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:46 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:38:46.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:46.941 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:46.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/726041406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:38:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:38:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:38:46.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:38:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/726041406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:38:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:38:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:38:46.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:46 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:38:47.044 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:47.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:47.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:47.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:47.226 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:47.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:47.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:47.362 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:47.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:47.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:47.496 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:47.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:47.598 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:47.643 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:47.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:47.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[37907]: pgmap v914: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:38:47.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:47.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:38:47.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[37907]: osdmap e418: 8 total, 8 up, 7 in 2024-03-24T11:38:47.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[37907]: 11.1 scrub starts 2024-03-24T11:38:47.663 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[29608]: pgmap v914: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[29608]: osdmap e418: 8 total, 8 up, 7 in 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[29608]: 11.1 scrub starts 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:38:47.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:47 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:38:47.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:47.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:47.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:47.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:47.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:47 smithi181 ceph-mon[33998]: pgmap v914: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-24T11:38:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:47 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:38:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:47 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:38:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:47 smithi181 ceph-mon[33998]: osdmap e418: 8 total, 8 up, 7 in 2024-03-24T11:38:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:47 smithi181 ceph-mon[33998]: 11.1 scrub starts 2024-03-24T11:38:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:47 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:38:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:47 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:38:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:47.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:47.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:47.921 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:47.985 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:48.004 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:38:48.004 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:38:48.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.072 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:48.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:48.173 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.214 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:48.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:48.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:48.339 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:48.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:48.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:48.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:48.604 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:48.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.732 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:48.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:48.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.863 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:48.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:48 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:38:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:48 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:38:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:48 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:38:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:48 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:38:48.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:38:48.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:38:48.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:38:48.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:38:48.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:38:48.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:38:48.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:38:48.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:48 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:38:48.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:48.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:48.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:49.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:49.126 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:49.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:49.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:49.257 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:49.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:49.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:49.387 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:49.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:49.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:38:49.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:49.514 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:49.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:49.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:49.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:49.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:49.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:49.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:49.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:49.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:49.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:49 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:38:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:49 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:38:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:49 smithi181 ceph-mon[33998]: pgmap v916: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-03-24T11:38:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:49 smithi181 ceph-mon[33998]: 11.1 scrub ok 2024-03-24T11:38:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:49 smithi181 ceph-mon[33998]: osdmap e419: 8 total, 8 up, 7 in 2024-03-24T11:38:49.933 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:49.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:49.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:49.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:38:49.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[37907]: pgmap v916: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[37907]: 11.1 scrub ok 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[37907]: osdmap e419: 8 total, 8 up, 7 in 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[29608]: pgmap v916: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-03-24T11:38:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[29608]: 11.1 scrub ok 2024-03-24T11:38:49.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:38:49.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:49 smithi179 ceph-mon[29608]: osdmap e419: 8 total, 8 up, 7 in 2024-03-24T11:38:50.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T11:38:50.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:50.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:50.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:50.165 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:50.224 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:50.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:50.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:50.349 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:50.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:50.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:50.479 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:50.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:50.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:50 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:38:50.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:50 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:38:50.576 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:50 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:38:50.581 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:50.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:50.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:50.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:50.751 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:50.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:50.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:50.885 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:50.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:50 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:38:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:50 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:38:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:50 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:38:50.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:50 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:38:50.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:50 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:38:50.923 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:50 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:38:50.986 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.027 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:51.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:51.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:51.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:51.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.296 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:51.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:51.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.421 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:51.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:51.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:51.570 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:51.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:51.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:51 smithi179 ceph-mon[29608]: pgmap v918: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 741 B/s rd, 0 B/s wr, 1.49k op/s 2024-03-24T11:38:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:51 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:38:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:51 smithi179 ceph-mon[29608]: 11.d scrub starts 2024-03-24T11:38:51.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:51 smithi179 ceph-mon[37907]: pgmap v918: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 741 B/s rd, 0 B/s wr, 1.49k op/s 2024-03-24T11:38:51.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:51 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:38:51.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:51 smithi179 ceph-mon[37907]: 11.d scrub starts 2024-03-24T11:38:51.801 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:51.848 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:51 smithi181 ceph-mon[33998]: pgmap v918: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 741 B/s rd, 0 B/s wr, 1.49k op/s 2024-03-24T11:38:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:51 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:38:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:51 smithi181 ceph-mon[33998]: 11.d scrub starts 2024-03-24T11:38:51.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:51.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:51.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:52.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:52.115 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:52.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:52.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:52.244 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:52.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:52.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:52.375 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:52.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:52.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:52.517 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:52.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:52.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:52.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:52.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:52.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:52.789 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:52.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:52.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:52.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:52 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:38:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:52 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:38:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:52 smithi181 ceph-mon[33998]: osdmap e420: 8 total, 8 up, 7 in 2024-03-24T11:38:52.920 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:52.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:52 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:38:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:52 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:38:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:52 smithi179 ceph-mon[29608]: osdmap e420: 8 total, 8 up, 7 in 2024-03-24T11:38:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:52 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:38:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:52 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:38:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:52 smithi179 ceph-mon[37907]: osdmap e420: 8 total, 8 up, 7 in 2024-03-24T11:38:53.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:53.050 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:53.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:53.072 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:53.151 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:53.159 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:38:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:53.213 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:53.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:53.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:53.347 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:53.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:53.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:53.523 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:53.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:53.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:53.670 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:53.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:38:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:38:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:38:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: pgmap v920: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 963 op/s 2024-03-24T11:38:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:38:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:38:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:38:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:38:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:38:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:38:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:38:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:38:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: pgmap v920: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 963 op/s 2024-03-24T11:38:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:38:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:38:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:38:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:38:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:53 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:38:53.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:53.813 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:53.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: pgmap v920: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 963 op/s 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:38:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:53 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:38:53.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:53.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:53.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:54.043 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:54.081 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:54.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:54.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:54.229 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:54.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:54.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:54.382 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:54.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:54.473 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:38:54.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:54.513 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:54.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:54.532 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:38:54.583 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:38:54.610 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:38:54.615 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:54.635 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:38:54.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:54.659 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:54.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:54.689 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:38:54.737 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:38:54.737 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:38:54.737 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:38:54.737 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 11.d scrub ok 2024-03-24T11:38:54.737 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:38:54.737 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:38:54.737 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: osdmap e421: 8 total, 8 up, 7 in 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3440193513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 11.d scrub ok 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:38:54.738 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:38:54.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:38:54.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:38:54.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:38:54.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:38:54.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: osdmap e421: 8 total, 8 up, 7 in 2024-03-24T11:38:54.739 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3440193513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:54.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:54.788 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:54.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:54.789 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:38:54.842 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:38:54.845 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:38:54.845 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:38:54.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:38:54.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:38:54.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:38:54.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:38:54.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:38:54.846 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:38:54.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 11.d scrub ok 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: osdmap e421: 8 total, 8 up, 7 in 2024-03-24T11:38:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3440193513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:38:54.917 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:54.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:55.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:55.067 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:55.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:55.107 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:38:55.111 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:38:55.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:55.207 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:55.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:55.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:55.336 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:55.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:55.385 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:38:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:38:55.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:55.466 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:55.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:55.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:55.607 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:55.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:55.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:55.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[29608]: 4.10 scrub starts 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[29608]: 4.10 scrub ok 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[29608]: pgmap v922: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[29608]: 11.e scrub starts 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[37907]: 4.10 scrub starts 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[37907]: 4.10 scrub ok 2024-03-24T11:38:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[37907]: pgmap v922: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-24T11:38:55.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[37907]: 11.e scrub starts 2024-03-24T11:38:55.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:38:55.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:55 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:38:55.741 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:55.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:55.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:55.876 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:55.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:38:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:55 smithi181 ceph-mon[33998]: 4.10 scrub starts 2024-03-24T11:38:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:55 smithi181 ceph-mon[33998]: 4.10 scrub ok 2024-03-24T11:38:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:55 smithi181 ceph-mon[33998]: pgmap v922: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-24T11:38:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:55 smithi181 ceph-mon[33998]: 11.e scrub starts 2024-03-24T11:38:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:55 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:38:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:55 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:38:55.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:56.007 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:56.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:56.140 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:56.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:56.276 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:56.378 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:56.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:56.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:56.542 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:56.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:56.676 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:56.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.816 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:56.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:56 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:38:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:56 smithi181 ceph-mon[33998]: from='client.25576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:38:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:56 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:38:56.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:56.956 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:56.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:56.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:56 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:38:56.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:56 smithi179 ceph-mon[29608]: from='client.25576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:38:56.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:56 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:38:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:56 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:38:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:56 smithi179 ceph-mon[37907]: from='client.25576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:38:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:56 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:38:57.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:57.091 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:57.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:57.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:57.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:57.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:57.330 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:57.363 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:57.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:57.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:57.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:57.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:57.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:57.622 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:57.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:57.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:38:57.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:57.753 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:57.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:38:57.854 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:38:57.892 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:57.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:38:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:57 smithi181 ceph-mon[33998]: 11.7 scrub starts 2024-03-24T11:38:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:57 smithi181 ceph-mon[33998]: pgmap v923: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 720 B/s rd, 0 B/s wr, 993 op/s 2024-03-24T11:38:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:57 smithi181 ceph-mon[33998]: 11.3 scrub starts 2024-03-24T11:38:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:57 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:38:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:57 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:38:57.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[29608]: 11.7 scrub starts 2024-03-24T11:38:57.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[29608]: pgmap v923: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 720 B/s rd, 0 B/s wr, 993 op/s 2024-03-24T11:38:57.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[29608]: 11.3 scrub starts 2024-03-24T11:38:57.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:38:57.943 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:38:57.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[37907]: 11.7 scrub starts 2024-03-24T11:38:57.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[37907]: pgmap v923: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 720 B/s rd, 0 B/s wr, 993 op/s 2024-03-24T11:38:57.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[37907]: 11.3 scrub starts 2024-03-24T11:38:57.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:38:57.944 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:57 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:38:57.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:58.037 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:58.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:58.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:58.188 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:58.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:58.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:38:58.316 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:58.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:38:58.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:58.452 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:58.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:58.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:58.589 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:58.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:58.691 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:58.718 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:58.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:58.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:58.857 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:58.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[29608]: 11.e scrub ok 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[29608]: 3.7 scrub starts 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[29608]: 3.7 scrub ok 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[29608]: 11.7 scrub ok 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[37907]: 11.e scrub ok 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[37907]: 3.7 scrub starts 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[37907]: 3.7 scrub ok 2024-03-24T11:38:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:58 smithi179 ceph-mon[37907]: 11.7 scrub ok 2024-03-24T11:38:58.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:58 smithi181 ceph-mon[33998]: 11.e scrub ok 2024-03-24T11:38:58.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:58 smithi181 ceph-mon[33998]: 3.7 scrub starts 2024-03-24T11:38:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:58 smithi181 ceph-mon[33998]: 3.7 scrub ok 2024-03-24T11:38:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:58 smithi181 ceph-mon[33998]: 11.7 scrub ok 2024-03-24T11:38:58.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:59.007 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:59.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:38:59.150 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:38:59.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:38:59.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:38:59.382 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.430 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:59.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:59.459 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:38:59.460 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:38:59.460 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:38:59.460 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:38:59.460 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:38:59.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:38:59.570 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:38:59.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:38:59.714 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:38:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:38:59.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:59.844 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:38:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:59 smithi181 ceph-mon[33998]: pgmap v924: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 814 op/s 2024-03-24T11:38:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:59 smithi181 ceph-mon[33998]: 11.3 scrub ok 2024-03-24T11:38:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:59 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:38:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:59 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:38:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:59 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:38:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:59 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:38:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:38:59 smithi181 ceph-mon[33998]: 11.6 scrub starts 2024-03-24T11:38:59.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:38:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[29608]: pgmap v924: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 814 op/s 2024-03-24T11:38:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[29608]: 11.3 scrub ok 2024-03-24T11:38:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:38:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:38:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:38:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:38:59.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[29608]: 11.6 scrub starts 2024-03-24T11:38:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[37907]: pgmap v924: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 814 op/s 2024-03-24T11:38:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[37907]: 11.3 scrub ok 2024-03-24T11:38:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:38:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:38:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:38:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:38:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:38:59 smithi179 ceph-mon[37907]: 11.6 scrub starts 2024-03-24T11:38:59.976 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:38:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:38:59.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:00.079 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:00.121 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:00.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:00.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:00.250 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:00.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:00.353 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:00.397 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:00.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:00.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:00.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:00.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:00.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:00.664 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:00.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:00.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:00.793 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:00.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:00.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:39:00.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:00.919 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:00.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:01.020 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:01.065 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:01.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:01.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:01.197 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:01.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:01.299 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:01.307 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":922,"stamp":"2024-03-24T11:38:54.446115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6427842934,"num_objects":98271,"num_object_clones":0,"num_object_copies":294813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98271,"num_whiteouts":0,"num_read":1645,"num_read_kb":1397,"num_write":143241,"num_write_kb":7698926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":331231,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71635,"ondisk_log_size":71635,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":24102548,"kb_used_data":18849876,"kb_used_omap":10,"kb_used_meta":5252149,"kb_avail":632113516,"statfs":{"total":671965249536,"available":647284240384,"internally_reserved":0,"allocated":19302273024,"data_stored":19299039524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":5378200684},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-525795328,"num_objects":-8023,"num_object_clones":0,"num_object_copies":-24069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8023,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8023,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001649"},"pg_stats":[{"pgid":"11.3","version":"420'8974","reported_seq":9060,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.592937+0000","last_change":"2024-03-24T11:36:51.539907+0000","last_active":"2024-03-24T11:38:53.592937+0000","last_peered":"2024-03-24T11:38:53.592937+0000","last_clean":"2024-03-24T11:38:53.592937+0000","last_became_active":"2024-03-24T11:36:51.539490+0000","last_became_peered":"2024-03-24T11:36:51.539490+0000","last_unstale":"2024-03-24T11:38:53.592937+0000","last_undegraded":"2024-03-24T11:38:53.592937+0000","last_fullsized":"2024-03-24T11:38:53.592937+0000","mapping_epoch":382,"log_start":"403'4400","ondisk_log_start":"403'4400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396886016,"num_objects":6056,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":480960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"213'20","reported_seq":459,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352300+0000","last_change":"2024-03-24T11:38:10.116849+0000","last_active":"2024-03-24T11:38:49.352300+0000","last_peered":"2024-03-24T11:38:49.352300+0000","last_clean":"2024-03-24T11:38:49.352300+0000","last_became_active":"2024-03-24T11:36:58.591623+0000","last_became_peered":"2024-03-24T11:36:58.591623+0000","last_unstale":"2024-03-24T11:38:49.352300+0000","last_undegraded":"2024-03-24T11:38:49.352300+0000","last_fullsized":"2024-03-24T11:38:49.352300+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:10.116699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.116699+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:20.360374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":770,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352114+0000","last_change":"2024-03-24T11:36:58.586904+0000","last_active":"2024-03-24T11:38:49.352114+0000","last_peered":"2024-03-24T11:38:49.352114+0000","last_clean":"2024-03-24T11:38:49.352114+0000","last_became_active":"2024-03-24T11:36:58.586602+0000","last_became_peered":"2024-03-24T11:36:58.586602+0000","last_unstale":"2024-03-24T11:38:49.352114+0000","last_undegraded":"2024-03-24T11:38:49.352114+0000","last_fullsized":"2024-03-24T11:38:49.352114+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:25.353054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:25.353054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105592+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"362'9","reported_seq":1056,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.601452+0000","last_change":"2024-03-24T11:37:37.413749+0000","last_active":"2024-03-24T11:38:51.601452+0000","last_peered":"2024-03-24T11:38:51.601452+0000","last_clean":"2024-03-24T11:38:51.601452+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:38:51.601452+0000","last_undegraded":"2024-03-24T11:38:51.601452+0000","last_fullsized":"2024-03-24T11:38:51.601452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'9","last_scrub_stamp":"2024-03-24T11:37:37.413561+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.413561+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:06.050090+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":84,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.604305+0000","last_change":"2024-03-24T11:38:01.792439+0000","last_active":"2024-03-24T11:38:51.604305+0000","last_peered":"2024-03-24T11:38:51.604305+0000","last_clean":"2024-03-24T11:38:51.604305+0000","last_became_active":"2024-03-24T11:36:58.590765+0000","last_became_peered":"2024-03-24T11:36:58.590765+0000","last_unstale":"2024-03-24T11:38:51.604305+0000","last_undegraded":"2024-03-24T11:38:51.604305+0000","last_fullsized":"2024-03-24T11:38:51.604305+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.792132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.792132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:24.724156+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":848,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.350802+0000","last_change":"2024-03-24T11:36:57.973025+0000","last_active":"2024-03-24T11:38:49.350802+0000","last_peered":"2024-03-24T11:38:49.350802+0000","last_clean":"2024-03-24T11:38:49.350802+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:38:49.350802+0000","last_undegraded":"2024-03-24T11:38:49.350802+0000","last_fullsized":"2024-03-24T11:38:49.350802+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:57.972898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:57.972898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"419'8766","reported_seq":8849,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:50.176997+0000","last_change":"2024-03-24T11:36:51.538337+0000","last_active":"2024-03-24T11:38:50.176997+0000","last_peered":"2024-03-24T11:38:50.176997+0000","last_clean":"2024-03-24T11:38:50.176997+0000","last_became_active":"2024-03-24T11:36:51.537990+0000","last_became_peered":"2024-03-24T11:36:51.537990+0000","last_unstale":"2024-03-24T11:38:50.176997+0000","last_undegraded":"2024-03-24T11:38:50.176997+0000","last_fullsized":"2024-03-24T11:38:50.176997+0000","mapping_epoch":382,"log_start":"405'4600","ondisk_log_start":"405'4600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408682496,"num_objects":6236,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8766,"num_write_kb":480064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"213'39","reported_seq":534,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.601248+0000","last_change":"2024-03-24T11:38:04.488873+0000","last_active":"2024-03-24T11:38:51.601248+0000","last_peered":"2024-03-24T11:38:51.601248+0000","last_clean":"2024-03-24T11:38:51.601248+0000","last_became_active":"2024-03-24T11:36:58.580651+0000","last_became_peered":"2024-03-24T11:36:58.580651+0000","last_unstale":"2024-03-24T11:38:51.601248+0000","last_undegraded":"2024-03-24T11:38:51.601248+0000","last_fullsized":"2024-03-24T11:38:51.601248+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:38:04.488661+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.488661+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:30.545546+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":833,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603035+0000","last_change":"2024-03-24T11:38:48.018756+0000","last_active":"2024-03-24T11:38:51.603035+0000","last_peered":"2024-03-24T11:38:51.603035+0000","last_clean":"2024-03-24T11:38:51.603035+0000","last_became_active":"2024-03-24T11:36:58.587097+0000","last_became_peered":"2024-03-24T11:36:58.587097+0000","last_unstale":"2024-03-24T11:38:51.603035+0000","last_undegraded":"2024-03-24T11:38:51.603035+0000","last_fullsized":"2024-03-24T11:38:51.603035+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:38:48.018617+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.018617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.449456+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":797,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603833+0000","last_change":"2024-03-24T11:37:04.724933+0000","last_active":"2024-03-24T11:38:51.603833+0000","last_peered":"2024-03-24T11:38:51.603833+0000","last_clean":"2024-03-24T11:38:51.603833+0000","last_became_active":"2024-03-24T11:36:58.595572+0000","last_became_peered":"2024-03-24T11:36:58.595572+0000","last_unstale":"2024-03-24T11:38:51.603833+0000","last_undegraded":"2024-03-24T11:38:51.603833+0000","last_fullsized":"2024-03-24T11:38:51.603833+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:04.724780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:37:04.724780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":98,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.356931+0000","last_change":"2024-03-24T11:36:58.582845+0000","last_active":"2024-03-24T11:38:49.356931+0000","last_peered":"2024-03-24T11:38:49.356931+0000","last_clean":"2024-03-24T11:38:49.356931+0000","last_became_active":"2024-03-24T11:36:58.582225+0000","last_became_peered":"2024-03-24T11:36:58.582225+0000","last_unstale":"2024-03-24T11:38:49.356931+0000","last_undegraded":"2024-03-24T11:38:49.356931+0000","last_fullsized":"2024-03-24T11:38:49.356931+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:26.128656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:26.128656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":956,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607315+0000","last_change":"2024-03-24T11:38:03.034263+0000","last_active":"2024-03-24T11:38:51.607315+0000","last_peered":"2024-03-24T11:38:51.607315+0000","last_clean":"2024-03-24T11:38:51.607315+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:38:51.607315+0000","last_undegraded":"2024-03-24T11:38:51.607315+0000","last_fullsized":"2024-03-24T11:38:51.607315+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:38:03.034131+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:38:03.034131+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:20.878170+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"420'9009","reported_seq":9168,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.574183+0000","last_change":"2024-03-24T11:38:48.786761+0000","last_active":"2024-03-24T11:38:53.574183+0000","last_peered":"2024-03-24T11:38:53.574183+0000","last_clean":"2024-03-24T11:38:53.574183+0000","last_became_active":"2024-03-24T11:36:51.529788+0000","last_became_peered":"2024-03-24T11:36:51.529788+0000","last_unstale":"2024-03-24T11:38:53.574183+0000","last_undegraded":"2024-03-24T11:38:53.574183+0000","last_fullsized":"2024-03-24T11:38:53.574183+0000","mapping_epoch":382,"log_start":"403'4400","ondisk_log_start":"403'4400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'8678","last_scrub_stamp":"2024-03-24T11:38:48.786544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.786544+0000","objects_scrubbed":6471,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:10.325229+0000","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398655488,"num_objects":6083,"num_object_clones":0,"num_object_copies":18249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"213'19","reported_seq":470,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603509+0000","last_change":"2024-03-24T11:37:31.892734+0000","last_active":"2024-03-24T11:38:51.603509+0000","last_peered":"2024-03-24T11:38:51.603509+0000","last_clean":"2024-03-24T11:38:51.603509+0000","last_became_active":"2024-03-24T11:36:58.587116+0000","last_became_peered":"2024-03-24T11:36:58.587116+0000","last_unstale":"2024-03-24T11:38:51.603509+0000","last_undegraded":"2024-03-24T11:38:51.603509+0000","last_fullsized":"2024-03-24T11:38:51.603509+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:37:31.892595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:31.892595+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:56.288241+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":84,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.601492+0000","last_change":"2024-03-24T11:38:47.247564+0000","last_active":"2024-03-24T11:38:51.601492+0000","last_peered":"2024-03-24T11:38:51.601492+0000","last_clean":"2024-03-24T11:38:51.601492+0000","last_became_active":"2024-03-24T11:36:58.585229+0000","last_became_peered":"2024-03-24T11:36:58.585229+0000","last_unstale":"2024-03-24T11:38:51.601492+0000","last_undegraded":"2024-03-24T11:38:51.601492+0000","last_fullsized":"2024-03-24T11:38:51.601492+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.247390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.247390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:17.162245+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":789,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608329+0000","last_change":"2024-03-24T11:37:38.108397+0000","last_active":"2024-03-24T11:38:51.608329+0000","last_peered":"2024-03-24T11:38:51.608329+0000","last_clean":"2024-03-24T11:38:51.608329+0000","last_became_active":"2024-03-24T11:36:58.575297+0000","last_became_peered":"2024-03-24T11:36:58.575297+0000","last_unstale":"2024-03-24T11:38:51.608329+0000","last_undegraded":"2024-03-24T11:38:51.608329+0000","last_fullsized":"2024-03-24T11:38:51.608329+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:38.108325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:37:38.108325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:04.811029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":752,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352041+0000","last_change":"2024-03-24T11:33:38.472649+0000","last_active":"2024-03-24T11:38:49.352041+0000","last_peered":"2024-03-24T11:38:49.352041+0000","last_clean":"2024-03-24T11:38:49.352041+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:38:49.352041+0000","last_undegraded":"2024-03-24T11:38:49.352041+0000","last_fullsized":"2024-03-24T11:38:49.352041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:38.472418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.472418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105618+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"420'8941","reported_seq":9028,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.405652+0000","last_change":"2024-03-24T11:36:51.545822+0000","last_active":"2024-03-24T11:38:53.405652+0000","last_peered":"2024-03-24T11:38:53.405652+0000","last_clean":"2024-03-24T11:38:53.405652+0000","last_became_active":"2024-03-24T11:36:51.545544+0000","last_became_peered":"2024-03-24T11:36:51.545544+0000","last_unstale":"2024-03-24T11:38:53.405652+0000","last_undegraded":"2024-03-24T11:38:53.405652+0000","last_fullsized":"2024-03-24T11:38:53.405652+0000","mapping_epoch":382,"log_start":"405'4700","ondisk_log_start":"405'4700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395509760,"num_objects":6035,"num_object_clones":0,"num_object_copies":18105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8941,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"213'34","reported_seq":867,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.609483+0000","last_change":"2024-03-24T11:36:58.618200+0000","last_active":"2024-03-24T11:38:51.609483+0000","last_peered":"2024-03-24T11:38:51.609483+0000","last_clean":"2024-03-24T11:38:51.609483+0000","last_became_active":"2024-03-24T11:36:58.594679+0000","last_became_peered":"2024-03-24T11:36:58.594679+0000","last_unstale":"2024-03-24T11:38:51.609483+0000","last_undegraded":"2024-03-24T11:38:51.609483+0000","last_fullsized":"2024-03-24T11:38:51.609483+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:36:56.684283+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:36:56.684283+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":773,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.351263+0000","last_change":"2024-03-24T11:37:01.005665+0000","last_active":"2024-03-24T11:38:49.351263+0000","last_peered":"2024-03-24T11:38:49.351263+0000","last_clean":"2024-03-24T11:38:49.351263+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:38:49.351263+0000","last_undegraded":"2024-03-24T11:38:49.351263+0000","last_fullsized":"2024-03-24T11:38:49.351263+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.005586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.005586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'9","reported_seq":144,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.903716+0000","last_change":"2024-03-24T11:38:51.903763+0000","last_active":"2024-03-24T11:38:51.903716+0000","last_peered":"2024-03-24T11:38:51.903716+0000","last_clean":"2024-03-24T11:38:51.903716+0000","last_became_active":"2024-03-24T11:36:58.574227+0000","last_became_peered":"2024-03-24T11:36:58.574227+0000","last_unstale":"2024-03-24T11:38:51.903716+0000","last_undegraded":"2024-03-24T11:38:51.903716+0000","last_fullsized":"2024-03-24T11:38:51.903716+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-03-24T11:38:51.903639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:51.903639+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.344849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":771,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607967+0000","last_change":"2024-03-24T11:36:51.385470+0000","last_active":"2024-03-24T11:38:51.607967+0000","last_peered":"2024-03-24T11:38:51.607967+0000","last_clean":"2024-03-24T11:38:51.607967+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:38:51.607967+0000","last_undegraded":"2024-03-24T11:38:51.607967+0000","last_fullsized":"2024-03-24T11:38:51.607967+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:51.385346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:51.385346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":947,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607263+0000","last_change":"2024-03-24T11:37:44.069186+0000","last_active":"2024-03-24T11:38:51.607263+0000","last_peered":"2024-03-24T11:38:51.607263+0000","last_clean":"2024-03-24T11:38:51.607263+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:38:51.607263+0000","last_undegraded":"2024-03-24T11:38:51.607263+0000","last_fullsized":"2024-03-24T11:38:51.607263+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:44.069073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.069073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:07.819002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"420'8929","reported_seq":9015,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.392837+0000","last_change":"2024-03-24T11:36:51.545109+0000","last_active":"2024-03-24T11:38:53.392837+0000","last_peered":"2024-03-24T11:38:53.392837+0000","last_clean":"2024-03-24T11:38:53.392837+0000","last_became_active":"2024-03-24T11:36:51.544798+0000","last_became_peered":"2024-03-24T11:36:51.544798+0000","last_unstale":"2024-03-24T11:38:53.392837+0000","last_undegraded":"2024-03-24T11:38:53.392837+0000","last_fullsized":"2024-03-24T11:38:53.392837+0000","mapping_epoch":382,"log_start":"405'4700","ondisk_log_start":"405'4700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394592256,"num_objects":6021,"num_object_clones":0,"num_object_copies":18063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8929,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":83,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.351706+0000","last_change":"2024-03-24T11:37:41.083113+0000","last_active":"2024-03-24T11:38:49.351706+0000","last_peered":"2024-03-24T11:38:49.351706+0000","last_clean":"2024-03-24T11:38:49.351706+0000","last_became_active":"2024-03-24T11:36:58.579874+0000","last_became_peered":"2024-03-24T11:36:58.579874+0000","last_unstale":"2024-03-24T11:38:49.351706+0000","last_undegraded":"2024-03-24T11:38:49.351706+0000","last_fullsized":"2024-03-24T11:38:49.351706+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.082984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.082984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106256+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"213'30","reported_seq":811,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.357410+0000","last_change":"2024-03-24T11:37:32.916715+0000","last_active":"2024-03-24T11:38:49.357410+0000","last_peered":"2024-03-24T11:38:49.357410+0000","last_clean":"2024-03-24T11:38:49.357410+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:38:49.357410+0000","last_undegraded":"2024-03-24T11:38:49.357410+0000","last_fullsized":"2024-03-24T11:38:49.357410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:37:32.916579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:32.916579+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.352266+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":765,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.356563+0000","last_change":"2024-03-24T11:33:36.491136+0000","last_active":"2024-03-24T11:38:49.356563+0000","last_peered":"2024-03-24T11:38:49.356563+0000","last_clean":"2024-03-24T11:38:49.356563+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:38:49.356563+0000","last_undegraded":"2024-03-24T11:38:49.356563+0000","last_fullsized":"2024-03-24T11:38:49.356563+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:33:36.491006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":74,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.357268+0000","last_change":"2024-03-24T11:36:58.575117+0000","last_active":"2024-03-24T11:38:49.357268+0000","last_peered":"2024-03-24T11:38:49.357268+0000","last_clean":"2024-03-24T11:38:49.357268+0000","last_became_active":"2024-03-24T11:36:58.574742+0000","last_became_peered":"2024-03-24T11:36:58.574742+0000","last_unstale":"2024-03-24T11:38:49.357268+0000","last_undegraded":"2024-03-24T11:38:49.357268+0000","last_fullsized":"2024-03-24T11:38:49.357268+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:30.982770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:33:30.982770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":391,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.606031+0000","last_change":"2024-03-24T11:37:01.308428+0000","last_active":"2024-03-24T11:38:51.606031+0000","last_peered":"2024-03-24T11:38:51.606031+0000","last_clean":"2024-03-24T11:38:51.606031+0000","last_became_active":"2024-03-24T11:37:01.307977+0000","last_became_peered":"2024-03-24T11:37:01.307977+0000","last_unstale":"2024-03-24T11:38:51.606031+0000","last_undegraded":"2024-03-24T11:38:51.606031+0000","last_fullsized":"2024-03-24T11:38:51.606031+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_clean_scrub_stamp":"2024-03-24T11:36:59.739800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"420'8858","reported_seq":8934,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.382779+0000","last_change":"2024-03-24T11:36:58.596375+0000","last_active":"2024-03-24T11:38:53.382779+0000","last_peered":"2024-03-24T11:38:53.382779+0000","last_clean":"2024-03-24T11:38:53.382779+0000","last_became_active":"2024-03-24T11:36:58.595829+0000","last_became_peered":"2024-03-24T11:36:58.595829+0000","last_unstale":"2024-03-24T11:38:53.382779+0000","last_undegraded":"2024-03-24T11:38:53.382779+0000","last_fullsized":"2024-03-24T11:38:53.382779+0000","mapping_epoch":387,"log_start":"405'4600","ondisk_log_start":"405'4600","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4258,"log_dups_size":0,"ondisk_log_size":4258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397148160,"num_objects":6060,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"388'9","reported_seq":1015,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.357564+0000","last_change":"2024-03-24T11:36:58.931284+0000","last_active":"2024-03-24T11:38:49.357564+0000","last_peered":"2024-03-24T11:38:49.357564+0000","last_clean":"2024-03-24T11:38:49.357564+0000","last_became_active":"2024-03-24T11:36:58.591546+0000","last_became_peered":"2024-03-24T11:36:58.591546+0000","last_unstale":"2024-03-24T11:38:49.357564+0000","last_undegraded":"2024-03-24T11:38:49.357564+0000","last_fullsized":"2024-03-24T11:38:49.357564+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:58.931185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:58.931185+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"213'28","reported_seq":477,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607269+0000","last_change":"2024-03-24T11:37:35.115468+0000","last_active":"2024-03-24T11:38:51.607269+0000","last_peered":"2024-03-24T11:38:51.607269+0000","last_clean":"2024-03-24T11:38:51.607269+0000","last_became_active":"2024-03-24T11:36:58.573987+0000","last_became_peered":"2024-03-24T11:36:58.573987+0000","last_unstale":"2024-03-24T11:38:51.607269+0000","last_undegraded":"2024-03-24T11:38:51.607269+0000","last_fullsized":"2024-03-24T11:38:51.607269+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:37:35.115336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.115336+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":811,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.619747+0000","last_change":"2024-03-24T11:38:09.054727+0000","last_active":"2024-03-24T11:38:51.619747+0000","last_peered":"2024-03-24T11:38:51.619747+0000","last_clean":"2024-03-24T11:38:51.619747+0000","last_became_active":"2024-03-24T11:36:58.594655+0000","last_became_peered":"2024-03-24T11:36:58.594655+0000","last_unstale":"2024-03-24T11:38:51.619747+0000","last_undegraded":"2024-03-24T11:38:51.619747+0000","last_fullsized":"2024-03-24T11:38:51.619747+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:09.054428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:09.054428+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:29.739709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":411,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603673+0000","last_change":"2024-03-24T11:38:10.119555+0000","last_active":"2024-03-24T11:38:51.603673+0000","last_peered":"2024-03-24T11:38:51.603673+0000","last_clean":"2024-03-24T11:38:51.603673+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:38:51.603673+0000","last_undegraded":"2024-03-24T11:38:51.603673+0000","last_fullsized":"2024-03-24T11:38:51.603673+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:10.119478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.119478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:19.763125+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":961,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.351870+0000","last_change":"2024-03-24T11:37:42.045811+0000","last_active":"2024-03-24T11:38:49.351870+0000","last_peered":"2024-03-24T11:38:49.351870+0000","last_clean":"2024-03-24T11:38:49.351870+0000","last_became_active":"2024-03-24T11:35:52.362283+0000","last_became_peered":"2024-03-24T11:35:52.362283+0000","last_unstale":"2024-03-24T11:38:49.351870+0000","last_undegraded":"2024-03-24T11:38:49.351870+0000","last_fullsized":"2024-03-24T11:38:49.351870+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:42.045652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.045652+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:03.550973+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"420'9144","reported_seq":9308,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.700773+0000","last_change":"2024-03-24T11:38:53.481972+0000","last_active":"2024-03-24T11:38:53.700773+0000","last_peered":"2024-03-24T11:38:53.700773+0000","last_clean":"2024-03-24T11:38:53.700773+0000","last_became_active":"2024-03-24T11:36:58.589952+0000","last_became_peered":"2024-03-24T11:36:58.589952+0000","last_unstale":"2024-03-24T11:38:53.700773+0000","last_undegraded":"2024-03-24T11:38:53.700773+0000","last_fullsized":"2024-03-24T11:38:53.700773+0000","mapping_epoch":387,"log_start":"403'4000","ondisk_log_start":"403'4000","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'9128","last_scrub_stamp":"2024-03-24T11:38:53.481477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.481477+0000","objects_scrubbed":6260,"log_size":5144,"log_dups_size":0,"ondisk_log_size":5144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.975145+0000","scrub_duration":2214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404946968,"num_objects":6180,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9144,"num_write_kb":490305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":1033,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.619502+0000","last_change":"2024-03-24T11:36:49.735543+0000","last_active":"2024-03-24T11:38:51.619502+0000","last_peered":"2024-03-24T11:38:51.619502+0000","last_clean":"2024-03-24T11:38:51.619502+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:38:51.619502+0000","last_undegraded":"2024-03-24T11:38:51.619502+0000","last_fullsized":"2024-03-24T11:38:51.619502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:36:49.735451+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:36:49.735451+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"213'20","reported_seq":844,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607086+0000","last_change":"2024-03-24T11:36:58.613794+0000","last_active":"2024-03-24T11:38:51.607086+0000","last_peered":"2024-03-24T11:38:51.607086+0000","last_clean":"2024-03-24T11:38:51.607086+0000","last_became_active":"2024-03-24T11:36:58.597641+0000","last_became_peered":"2024-03-24T11:36:58.597641+0000","last_unstale":"2024-03-24T11:38:51.607086+0000","last_undegraded":"2024-03-24T11:38:51.607086+0000","last_fullsized":"2024-03-24T11:38:51.607086+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:36:03.939287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:03.939287+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":803,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608156+0000","last_change":"2024-03-24T11:37:45.089273+0000","last_active":"2024-03-24T11:38:51.608156+0000","last_peered":"2024-03-24T11:38:51.608156+0000","last_clean":"2024-03-24T11:38:51.608156+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:38:51.608156+0000","last_undegraded":"2024-03-24T11:38:51.608156+0000","last_fullsized":"2024-03-24T11:38:51.608156+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:45.089152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_clean_scrub_stamp":"2024-03-24T11:37:45.089152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":773,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.610598+0000","last_change":"2024-03-24T11:36:45.718958+0000","last_active":"2024-03-24T11:38:51.610598+0000","last_peered":"2024-03-24T11:38:51.610598+0000","last_clean":"2024-03-24T11:38:51.610598+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:38:51.610598+0000","last_undegraded":"2024-03-24T11:38:51.610598+0000","last_fullsized":"2024-03-24T11:38:51.610598+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:36:45.718846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:45.718846+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":984,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603704+0000","last_change":"2024-03-24T11:37:41.909224+0000","last_active":"2024-03-24T11:38:51.603704+0000","last_peered":"2024-03-24T11:38:51.603704+0000","last_clean":"2024-03-24T11:38:51.603704+0000","last_became_active":"2024-03-24T11:35:39.758022+0000","last_became_peered":"2024-03-24T11:35:39.758022+0000","last_unstale":"2024-03-24T11:38:51.603704+0000","last_undegraded":"2024-03-24T11:38:51.603704+0000","last_fullsized":"2024-03-24T11:38:51.603704+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:41.909070+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.909070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.644204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"420'8885","reported_seq":8977,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.376630+0000","last_change":"2024-03-24T11:36:58.590495+0000","last_active":"2024-03-24T11:38:53.376630+0000","last_peered":"2024-03-24T11:38:53.376630+0000","last_clean":"2024-03-24T11:38:53.376630+0000","last_became_active":"2024-03-24T11:36:58.590179+0000","last_became_peered":"2024-03-24T11:36:58.590179+0000","last_unstale":"2024-03-24T11:38:53.376630+0000","last_undegraded":"2024-03-24T11:38:53.376630+0000","last_fullsized":"2024-03-24T11:38:53.376630+0000","mapping_epoch":387,"log_start":"405'4700","ondisk_log_start":"405'4700","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396689408,"num_objects":6053,"num_object_clones":0,"num_object_copies":18159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":808,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603219+0000","last_change":"2024-03-24T11:37:43.909192+0000","last_active":"2024-03-24T11:38:51.603219+0000","last_peered":"2024-03-24T11:38:51.603219+0000","last_clean":"2024-03-24T11:38:51.603219+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:38:51.603219+0000","last_undegraded":"2024-03-24T11:38:51.603219+0000","last_fullsized":"2024-03-24T11:38:51.603219+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.909062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.909062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:01.175526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"213'29","reported_seq":839,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.619070+0000","last_change":"2024-03-24T11:38:04.942863+0000","last_active":"2024-03-24T11:38:51.619070+0000","last_peered":"2024-03-24T11:38:51.619070+0000","last_clean":"2024-03-24T11:38:51.619070+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:38:51.619070+0000","last_undegraded":"2024-03-24T11:38:51.619070+0000","last_fullsized":"2024-03-24T11:38:51.619070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:38:04.942715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.942715+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:34.510965+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":808,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603622+0000","last_change":"2024-03-24T11:37:49.656241+0000","last_active":"2024-03-24T11:38:51.603622+0000","last_peered":"2024-03-24T11:38:51.603622+0000","last_clean":"2024-03-24T11:38:51.603622+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:38:51.603622+0000","last_undegraded":"2024-03-24T11:38:51.603622+0000","last_fullsized":"2024-03-24T11:38:51.603622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:49.656104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:49.656104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.626409+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":788,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602812+0000","last_change":"2024-03-24T11:37:45.747922+0000","last_active":"2024-03-24T11:38:51.602812+0000","last_peered":"2024-03-24T11:38:51.602812+0000","last_clean":"2024-03-24T11:38:51.602812+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:38:51.602812+0000","last_undegraded":"2024-03-24T11:38:51.602812+0000","last_fullsized":"2024-03-24T11:38:51.602812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:45.747799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:45.747799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:58.909782+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":984,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603623+0000","last_change":"2024-03-24T11:37:46.878302+0000","last_active":"2024-03-24T11:38:51.603623+0000","last_peered":"2024-03-24T11:38:51.603623+0000","last_clean":"2024-03-24T11:38:51.603623+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:38:51.603623+0000","last_undegraded":"2024-03-24T11:38:51.603623+0000","last_fullsized":"2024-03-24T11:38:51.603623+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:46.878216+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:33:32.751865+0000","last_clean_scrub_stamp":"2024-03-24T11:37:46.878216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"419'8863","reported_seq":8946,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:50.122473+0000","last_change":"2024-03-24T11:36:51.535382+0000","last_active":"2024-03-24T11:38:50.122473+0000","last_peered":"2024-03-24T11:38:50.122473+0000","last_clean":"2024-03-24T11:38:50.122473+0000","last_became_active":"2024-03-24T11:36:51.534255+0000","last_became_peered":"2024-03-24T11:36:51.534255+0000","last_unstale":"2024-03-24T11:38:50.122473+0000","last_undegraded":"2024-03-24T11:38:50.122473+0000","last_fullsized":"2024-03-24T11:38:50.122473+0000","mapping_epoch":382,"log_start":"405'4700","ondisk_log_start":"405'4700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416743424,"num_objects":6359,"num_object_clones":0,"num_object_copies":19077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":784,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608364+0000","last_change":"2024-03-24T11:37:43.142179+0000","last_active":"2024-03-24T11:38:51.608364+0000","last_peered":"2024-03-24T11:38:51.608364+0000","last_clean":"2024-03-24T11:38:51.608364+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:38:51.608364+0000","last_undegraded":"2024-03-24T11:38:51.608364+0000","last_fullsized":"2024-03-24T11:38:51.608364+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.142063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.142063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"213'24","reported_seq":333,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607961+0000","last_change":"2024-03-24T11:37:02.374246+0000","last_active":"2024-03-24T11:38:51.607961+0000","last_peered":"2024-03-24T11:38:51.607961+0000","last_clean":"2024-03-24T11:38:51.607961+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:38:51.607961+0000","last_undegraded":"2024-03-24T11:38:51.607961+0000","last_fullsized":"2024-03-24T11:38:51.607961+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:37:02.374142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:02.374142+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":944,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:52.248721+0000","last_change":"2024-03-24T11:37:03.767302+0000","last_active":"2024-03-24T11:38:52.248721+0000","last_peered":"2024-03-24T11:38:52.248721+0000","last_clean":"2024-03-24T11:38:52.248721+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:38:52.248721+0000","last_undegraded":"2024-03-24T11:38:52.248721+0000","last_fullsized":"2024-03-24T11:38:52.248721+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:37:03.767151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:37:03.767151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":503,"num_read_kb":498,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":801,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.615857+0000","last_change":"2024-03-24T11:36:58.573150+0000","last_active":"2024-03-24T11:38:51.615857+0000","last_peered":"2024-03-24T11:38:51.615857+0000","last_clean":"2024-03-24T11:38:51.615857+0000","last_became_active":"2024-03-24T11:36:58.572807+0000","last_became_peered":"2024-03-24T11:36:58.572807+0000","last_unstale":"2024-03-24T11:38:51.615857+0000","last_undegraded":"2024-03-24T11:38:51.615857+0000","last_fullsized":"2024-03-24T11:38:51.615857+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:56.781495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:36:56.781495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":796,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603350+0000","last_change":"2024-03-24T11:38:05.656269+0000","last_active":"2024-03-24T11:38:51.603350+0000","last_peered":"2024-03-24T11:38:51.603350+0000","last_clean":"2024-03-24T11:38:51.603350+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:38:51.603350+0000","last_undegraded":"2024-03-24T11:38:51.603350+0000","last_fullsized":"2024-03-24T11:38:51.603350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.656146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.656146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:28.915290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":995,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607183+0000","last_change":"2024-03-24T11:37:01.308329+0000","last_active":"2024-03-24T11:38:51.607183+0000","last_peered":"2024-03-24T11:38:51.607183+0000","last_clean":"2024-03-24T11:38:51.607183+0000","last_became_active":"2024-03-24T11:37:01.307925+0000","last_became_peered":"2024-03-24T11:37:01.307925+0000","last_unstale":"2024-03-24T11:38:51.607183+0000","last_undegraded":"2024-03-24T11:38:51.607183+0000","last_fullsized":"2024-03-24T11:38:51.607183+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:49.592471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:49.592471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"419'8703","reported_seq":8786,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:50.147861+0000","last_change":"2024-03-24T11:36:51.533861+0000","last_active":"2024-03-24T11:38:50.147861+0000","last_peered":"2024-03-24T11:38:50.147861+0000","last_clean":"2024-03-24T11:38:50.147861+0000","last_became_active":"2024-03-24T11:36:51.533003+0000","last_became_peered":"2024-03-24T11:36:51.533003+0000","last_unstale":"2024-03-24T11:38:50.147861+0000","last_undegraded":"2024-03-24T11:38:50.147861+0000","last_fullsized":"2024-03-24T11:38:50.147861+0000","mapping_epoch":382,"log_start":"404'4500","ondisk_log_start":"404'4500","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409010176,"num_objects":6241,"num_object_clones":0,"num_object_copies":18723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8703,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":411,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602292+0000","last_change":"2024-03-24T11:38:05.438477+0000","last_active":"2024-03-24T11:38:51.602292+0000","last_peered":"2024-03-24T11:38:51.602292+0000","last_clean":"2024-03-24T11:38:51.602292+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:38:51.602292+0000","last_undegraded":"2024-03-24T11:38:51.602292+0000","last_fullsized":"2024-03-24T11:38:51.602292+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.438380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.438380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:33.739925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"388'37","reported_seq":1124,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603550+0000","last_change":"2024-03-24T11:37:52.944624+0000","last_active":"2024-03-24T11:38:51.603550+0000","last_peered":"2024-03-24T11:38:51.603550+0000","last_clean":"2024-03-24T11:38:51.603550+0000","last_became_active":"2024-03-24T11:36:58.574117+0000","last_became_peered":"2024-03-24T11:36:58.574117+0000","last_unstale":"2024-03-24T11:38:51.603550+0000","last_undegraded":"2024-03-24T11:38:51.603550+0000","last_fullsized":"2024-03-24T11:38:51.603550+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-03-24T11:37:52.944533+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.944533+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:16.899416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":779,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352421+0000","last_change":"2024-03-24T11:37:35.979186+0000","last_active":"2024-03-24T11:38:49.352421+0000","last_peered":"2024-03-24T11:38:49.352421+0000","last_clean":"2024-03-24T11:38:49.352421+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:38:49.352421+0000","last_undegraded":"2024-03-24T11:38:49.352421+0000","last_fullsized":"2024-03-24T11:38:49.352421+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.978638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.978638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106231+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":732,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.610702+0000","last_change":"2024-03-24T11:37:02.957037+0000","last_active":"2024-03-24T11:38:51.610702+0000","last_peered":"2024-03-24T11:38:51.610702+0000","last_clean":"2024-03-24T11:38:51.610702+0000","last_became_active":"2024-03-24T11:35:35.730443+0000","last_became_peered":"2024-03-24T11:35:35.730443+0000","last_unstale":"2024-03-24T11:38:51.610702+0000","last_undegraded":"2024-03-24T11:38:51.610702+0000","last_fullsized":"2024-03-24T11:38:51.610702+0000","mapping_epoch":237,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:02.956947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:02.956947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":802,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608210+0000","last_change":"2024-03-24T11:38:02.291361+0000","last_active":"2024-03-24T11:38:51.608210+0000","last_peered":"2024-03-24T11:38:51.608210+0000","last_clean":"2024-03-24T11:38:51.608210+0000","last_became_active":"2024-03-24T11:36:58.586913+0000","last_became_peered":"2024-03-24T11:36:58.586913+0000","last_unstale":"2024-03-24T11:38:51.608210+0000","last_undegraded":"2024-03-24T11:38:51.608210+0000","last_fullsized":"2024-03-24T11:38:51.608210+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:02.291240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:38:02.291240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.313586+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"419'8782","reported_seq":8865,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:50.148256+0000","last_change":"2024-03-24T11:36:51.530980+0000","last_active":"2024-03-24T11:38:50.148256+0000","last_peered":"2024-03-24T11:38:50.148256+0000","last_clean":"2024-03-24T11:38:50.148256+0000","last_became_active":"2024-03-24T11:36:51.530664+0000","last_became_peered":"2024-03-24T11:36:51.530664+0000","last_unstale":"2024-03-24T11:38:50.148256+0000","last_undegraded":"2024-03-24T11:38:50.148256+0000","last_fullsized":"2024-03-24T11:38:50.148256+0000","mapping_epoch":382,"log_start":"405'4600","ondisk_log_start":"405'4600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411435008,"num_objects":6278,"num_object_clones":0,"num_object_copies":18834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":808,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.611463+0000","last_change":"2024-03-24T11:37:57.458128+0000","last_active":"2024-03-24T11:38:51.611463+0000","last_peered":"2024-03-24T11:38:51.611463+0000","last_clean":"2024-03-24T11:38:51.611463+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:38:51.611463+0000","last_undegraded":"2024-03-24T11:38:51.611463+0000","last_fullsized":"2024-03-24T11:38:51.611463+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:57.457058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:57.457058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:11.932433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"213'17","reported_seq":812,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.606808+0000","last_change":"2024-03-24T11:37:39.487537+0000","last_active":"2024-03-24T11:38:51.606808+0000","last_peered":"2024-03-24T11:38:51.606808+0000","last_clean":"2024-03-24T11:38:51.606808+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:38:51.606808+0000","last_undegraded":"2024-03-24T11:38:51.606808+0000","last_fullsized":"2024-03-24T11:38:51.606808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'17","last_scrub_stamp":"2024-03-24T11:37:39.487379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:39.487379+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":825,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.601583+0000","last_change":"2024-03-24T11:37:59.474830+0000","last_active":"2024-03-24T11:38:51.601583+0000","last_peered":"2024-03-24T11:38:51.601583+0000","last_clean":"2024-03-24T11:38:51.601583+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:38:51.601583+0000","last_undegraded":"2024-03-24T11:38:51.601583+0000","last_fullsized":"2024-03-24T11:38:51.601583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:59.474714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:37:59.474714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:25.516288+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":808,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602401+0000","last_change":"2024-03-24T11:37:53.683206+0000","last_active":"2024-03-24T11:38:51.602401+0000","last_peered":"2024-03-24T11:38:51.602401+0000","last_clean":"2024-03-24T11:38:51.602401+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:38:51.602401+0000","last_undegraded":"2024-03-24T11:38:51.602401+0000","last_fullsized":"2024-03-24T11:38:51.602401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:53.683090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:53.683090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:10.088022+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":403,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603597+0000","last_change":"2024-03-24T11:36:55.799282+0000","last_active":"2024-03-24T11:38:51.603597+0000","last_peered":"2024-03-24T11:38:51.603597+0000","last_clean":"2024-03-24T11:38:51.603597+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:38:51.603597+0000","last_undegraded":"2024-03-24T11:38:51.603597+0000","last_fullsized":"2024-03-24T11:38:51.603597+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:55.799186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:36:55.799186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":750,"reported_epoch":420,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-24T11:38:51.617528+0000","last_change":"2024-03-24T11:38:29.389080+0000","last_active":"2024-03-24T11:38:51.617528+0000","last_peered":"2024-03-24T11:38:51.617528+0000","last_clean":"2024-03-24T11:36:55.645795+0000","last_became_active":"2024-03-24T11:36:59.373164+0000","last_became_peered":"2024-03-24T11:36:59.373164+0000","last_unstale":"2024-03-24T11:38:51.617528+0000","last_undegraded":"2024-03-24T11:38:51.617528+0000","last_fullsized":"2024-03-24T11:38:51.617528+0000","mapping_epoch":388,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:00.701398+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.701398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,4],"avail_no_missing":["0","2","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"419'8792","reported_seq":8875,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:50.161977+0000","last_change":"2024-03-24T11:36:51.532092+0000","last_active":"2024-03-24T11:38:50.161977+0000","last_peered":"2024-03-24T11:38:50.161977+0000","last_clean":"2024-03-24T11:38:50.161977+0000","last_became_active":"2024-03-24T11:36:51.531785+0000","last_became_peered":"2024-03-24T11:36:51.531785+0000","last_unstale":"2024-03-24T11:38:50.161977+0000","last_undegraded":"2024-03-24T11:38:50.161977+0000","last_fullsized":"2024-03-24T11:38:50.161977+0000","mapping_epoch":382,"log_start":"404'4600","ondisk_log_start":"404'4600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413138944,"num_objects":6304,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8792,"num_write_kb":483072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":790,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.617050+0000","last_change":"2024-03-24T11:37:51.788499+0000","last_active":"2024-03-24T11:38:51.617050+0000","last_peered":"2024-03-24T11:38:51.617050+0000","last_clean":"2024-03-24T11:38:51.617050+0000","last_became_active":"2024-03-24T11:36:58.594154+0000","last_became_peered":"2024-03-24T11:36:58.594154+0000","last_unstale":"2024-03-24T11:38:51.617050+0000","last_undegraded":"2024-03-24T11:38:51.617050+0000","last_fullsized":"2024-03-24T11:38:51.617050+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.788360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.788360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:56.964510+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"401'41","reported_seq":1130,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603697+0000","last_change":"2024-03-24T11:37:44.707841+0000","last_active":"2024-03-24T11:38:51.603697+0000","last_peered":"2024-03-24T11:38:51.603697+0000","last_clean":"2024-03-24T11:38:51.603697+0000","last_became_active":"2024-03-24T11:37:33.888447+0000","last_became_peered":"2024-03-24T11:37:33.888447+0000","last_unstale":"2024-03-24T11:38:51.603697+0000","last_undegraded":"2024-03-24T11:38:51.603697+0000","last_fullsized":"2024-03-24T11:38:51.603697+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'41","last_scrub_stamp":"2024-03-24T11:37:44.707693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.707693+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:57.814576+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":84,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.618969+0000","last_change":"2024-03-24T11:38:01.964312+0000","last_active":"2024-03-24T11:38:51.618969+0000","last_peered":"2024-03-24T11:38:51.618969+0000","last_clean":"2024-03-24T11:38:51.618969+0000","last_became_active":"2024-03-24T11:36:58.591467+0000","last_became_peered":"2024-03-24T11:36:58.591467+0000","last_unstale":"2024-03-24T11:38:51.618969+0000","last_undegraded":"2024-03-24T11:38:51.618969+0000","last_fullsized":"2024-03-24T11:38:51.618969+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.964189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.348439+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":803,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607475+0000","last_change":"2024-03-24T11:37:03.354928+0000","last_active":"2024-03-24T11:38:51.607475+0000","last_peered":"2024-03-24T11:38:51.607475+0000","last_clean":"2024-03-24T11:38:51.607475+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:38:51.607475+0000","last_undegraded":"2024-03-24T11:38:51.607475+0000","last_fullsized":"2024-03-24T11:38:51.607475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:37:03.354797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:37:03.354797+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":759,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.356194+0000","last_change":"2024-03-24T11:36:58.579018+0000","last_active":"2024-03-24T11:38:49.356194+0000","last_peered":"2024-03-24T11:38:49.356194+0000","last_clean":"2024-03-24T11:38:49.356194+0000","last_became_active":"2024-03-24T11:36:58.578703+0000","last_became_peered":"2024-03-24T11:36:58.578703+0000","last_unstale":"2024-03-24T11:38:49.356194+0000","last_undegraded":"2024-03-24T11:38:49.356194+0000","last_fullsized":"2024-03-24T11:38:49.356194+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:35.480687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:35.480687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105463+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":978,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607274+0000","last_change":"2024-03-24T11:37:55.105164+0000","last_active":"2024-03-24T11:38:51.607274+0000","last_peered":"2024-03-24T11:38:51.607274+0000","last_clean":"2024-03-24T11:38:51.607274+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:38:51.607274+0000","last_undegraded":"2024-03-24T11:38:51.607274+0000","last_fullsized":"2024-03-24T11:38:51.607274+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:55.105007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.105007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.410544+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"420'8898","reported_seq":8984,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.584327+0000","last_change":"2024-03-24T11:36:51.539627+0000","last_active":"2024-03-24T11:38:53.584327+0000","last_peered":"2024-03-24T11:38:53.584327+0000","last_clean":"2024-03-24T11:38:53.584327+0000","last_became_active":"2024-03-24T11:36:51.539297+0000","last_became_peered":"2024-03-24T11:36:51.539297+0000","last_unstale":"2024-03-24T11:38:53.584327+0000","last_undegraded":"2024-03-24T11:38:53.584327+0000","last_fullsized":"2024-03-24T11:38:53.584327+0000","mapping_epoch":382,"log_start":"403'4300","ondisk_log_start":"403'4300","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393084928,"num_objects":5998,"num_object_clones":0,"num_object_copies":17994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8898,"num_write_kb":476672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"213'15","reported_seq":828,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.619398+0000","last_change":"2024-03-24T11:36:58.617099+0000","last_active":"2024-03-24T11:38:51.619398+0000","last_peered":"2024-03-24T11:38:51.619398+0000","last_clean":"2024-03-24T11:38:51.619398+0000","last_became_active":"2024-03-24T11:36:58.590879+0000","last_became_peered":"2024-03-24T11:36:58.590879+0000","last_unstale":"2024-03-24T11:38:51.619398+0000","last_undegraded":"2024-03-24T11:38:51.619398+0000","last_fullsized":"2024-03-24T11:38:51.619398+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:36:54.778072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:54.778072+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":805,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607815+0000","last_change":"2024-03-24T11:38:05.022910+0000","last_active":"2024-03-24T11:38:51.607815+0000","last_peered":"2024-03-24T11:38:51.607815+0000","last_clean":"2024-03-24T11:38:51.607815+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:38:51.607815+0000","last_undegraded":"2024-03-24T11:38:51.607815+0000","last_fullsized":"2024-03-24T11:38:51.607815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:05.022785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.022785+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:20.167706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":76,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.357390+0000","last_change":"2024-03-24T11:36:58.577922+0000","last_active":"2024-03-24T11:38:49.357390+0000","last_peered":"2024-03-24T11:38:49.357390+0000","last_clean":"2024-03-24T11:38:49.357390+0000","last_became_active":"2024-03-24T11:36:58.576200+0000","last_became_peered":"2024-03-24T11:36:58.576200+0000","last_unstale":"2024-03-24T11:38:49.357390+0000","last_undegraded":"2024-03-24T11:38:49.357390+0000","last_fullsized":"2024-03-24T11:38:49.357390+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:37.858877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:37.858877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.617351+0000","last_change":"2024-03-24T11:37:35.857267+0000","last_active":"2024-03-24T11:38:51.617351+0000","last_peered":"2024-03-24T11:38:51.617351+0000","last_clean":"2024-03-24T11:38:51.617351+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:38:51.617351+0000","last_undegraded":"2024-03-24T11:38:51.617351+0000","last_fullsized":"2024-03-24T11:38:51.617351+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.856841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.856841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"420'9039","reported_seq":9125,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.639087+0000","last_change":"2024-03-24T11:36:51.541061+0000","last_active":"2024-03-24T11:38:53.639087+0000","last_peered":"2024-03-24T11:38:53.639087+0000","last_clean":"2024-03-24T11:38:53.639087+0000","last_became_active":"2024-03-24T11:36:51.540802+0000","last_became_peered":"2024-03-24T11:36:51.540802+0000","last_unstale":"2024-03-24T11:38:53.639087+0000","last_undegraded":"2024-03-24T11:38:53.639087+0000","last_fullsized":"2024-03-24T11:38:53.639087+0000","mapping_epoch":382,"log_start":"403'4400","ondisk_log_start":"403'4400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401670144,"num_objects":6129,"num_object_clones":0,"num_object_copies":18387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"213'44","reported_seq":868,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603274+0000","last_change":"2024-03-24T11:37:34.852381+0000","last_active":"2024-03-24T11:38:51.603274+0000","last_peered":"2024-03-24T11:38:51.603274+0000","last_clean":"2024-03-24T11:38:51.603274+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:38:51.603274+0000","last_undegraded":"2024-03-24T11:38:51.603274+0000","last_fullsized":"2024-03-24T11:38:51.603274+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:37:34.852224+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:37:34.852224+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:59.543798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":403,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608314+0000","last_change":"2024-03-24T11:38:46.876895+0000","last_active":"2024-03-24T11:38:51.608314+0000","last_peered":"2024-03-24T11:38:51.608314+0000","last_clean":"2024-03-24T11:38:51.608314+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:38:51.608314+0000","last_undegraded":"2024-03-24T11:38:51.608314+0000","last_fullsized":"2024-03-24T11:38:51.608314+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:46.876805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:46.876805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:04.648439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":807,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602314+0000","last_change":"2024-03-24T11:37:52.553515+0000","last_active":"2024-03-24T11:38:51.602314+0000","last_peered":"2024-03-24T11:38:51.602314+0000","last_clean":"2024-03-24T11:38:51.602314+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:38:51.602314+0000","last_undegraded":"2024-03-24T11:38:51.602314+0000","last_fullsized":"2024-03-24T11:38:51.602314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:52.553402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.553402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:03.813860+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":101,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.350482+0000","last_change":"2024-03-24T11:38:12.123596+0000","last_active":"2024-03-24T11:38:49.350482+0000","last_peered":"2024-03-24T11:38:49.350482+0000","last_clean":"2024-03-24T11:38:49.350482+0000","last_became_active":"2024-03-24T11:36:42.992990+0000","last_became_peered":"2024-03-24T11:36:42.992990+0000","last_unstale":"2024-03-24T11:38:49.350482+0000","last_undegraded":"2024-03-24T11:38:49.350482+0000","last_fullsized":"2024-03-24T11:38:49.350482+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.123323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:40.144012+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"420'9023","reported_seq":9109,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.593331+0000","last_change":"2024-03-24T11:36:51.540130+0000","last_active":"2024-03-24T11:38:53.593331+0000","last_peered":"2024-03-24T11:38:53.593331+0000","last_clean":"2024-03-24T11:38:53.593331+0000","last_became_active":"2024-03-24T11:36:51.539372+0000","last_became_peered":"2024-03-24T11:36:51.539372+0000","last_unstale":"2024-03-24T11:38:53.593331+0000","last_undegraded":"2024-03-24T11:38:53.593331+0000","last_fullsized":"2024-03-24T11:38:53.593331+0000","mapping_epoch":382,"log_start":"405'4700","ondisk_log_start":"405'4700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399048704,"num_objects":6089,"num_object_clones":0,"num_object_copies":18267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":483584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":103,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.612154+0000","last_change":"2024-03-24T11:38:11.115502+0000","last_active":"2024-03-24T11:38:51.612154+0000","last_peered":"2024-03-24T11:38:51.612154+0000","last_clean":"2024-03-24T11:38:51.612154+0000","last_became_active":"2024-03-24T11:36:42.470376+0000","last_became_peered":"2024-03-24T11:36:42.470376+0000","last_unstale":"2024-03-24T11:38:51.612154+0000","last_undegraded":"2024-03-24T11:38:51.612154+0000","last_fullsized":"2024-03-24T11:38:51.612154+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:11.115390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.115390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:37.708874+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"213'18","reported_seq":471,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:52.571003+0000","last_change":"2024-03-24T11:38:52.571057+0000","last_active":"2024-03-24T11:38:52.571003+0000","last_peered":"2024-03-24T11:38:52.571003+0000","last_clean":"2024-03-24T11:38:52.571003+0000","last_became_active":"2024-03-24T11:36:58.593181+0000","last_became_peered":"2024-03-24T11:36:58.593181+0000","last_unstale":"2024-03-24T11:38:52.571003+0000","last_undegraded":"2024-03-24T11:38:52.571003+0000","last_fullsized":"2024-03-24T11:38:52.571003+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:38:52.570919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.570919+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.066756+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":86,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.610499+0000","last_change":"2024-03-24T11:37:47.674201+0000","last_active":"2024-03-24T11:38:51.610499+0000","last_peered":"2024-03-24T11:38:51.610499+0000","last_clean":"2024-03-24T11:38:51.610499+0000","last_became_active":"2024-03-24T11:36:58.598204+0000","last_became_peered":"2024-03-24T11:36:58.598204+0000","last_unstale":"2024-03-24T11:38:51.610499+0000","last_undegraded":"2024-03-24T11:38:51.610499+0000","last_fullsized":"2024-03-24T11:38:51.610499+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:47.673879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:37:47.673879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:05.076109+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":752,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352439+0000","last_change":"2024-03-24T11:33:42.747143+0000","last_active":"2024-03-24T11:38:49.352439+0000","last_peered":"2024-03-24T11:38:49.352439+0000","last_clean":"2024-03-24T11:38:49.352439+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:38:49.352439+0000","last_undegraded":"2024-03-24T11:38:49.352439+0000","last_fullsized":"2024-03-24T11:38:49.352439+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.746752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.746752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105687+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"420'8913","reported_seq":9005,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.649418+0000","last_change":"2024-03-24T11:36:58.587646+0000","last_active":"2024-03-24T11:38:53.649418+0000","last_peered":"2024-03-24T11:38:53.649418+0000","last_clean":"2024-03-24T11:38:53.649418+0000","last_became_active":"2024-03-24T11:36:58.587355+0000","last_became_peered":"2024-03-24T11:36:58.587355+0000","last_unstale":"2024-03-24T11:38:53.649418+0000","last_undegraded":"2024-03-24T11:38:53.649418+0000","last_fullsized":"2024-03-24T11:38:53.649418+0000","mapping_epoch":387,"log_start":"403'4200","ondisk_log_start":"403'4200","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4713,"log_dups_size":0,"ondisk_log_size":4713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390004736,"num_objects":5951,"num_object_clones":0,"num_object_copies":17853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8913,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"388'11","reported_seq":547,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603204+0000","last_change":"2024-03-24T11:37:58.943704+0000","last_active":"2024-03-24T11:38:51.603204+0000","last_peered":"2024-03-24T11:38:51.603204+0000","last_clean":"2024-03-24T11:38:51.603204+0000","last_became_active":"2024-03-24T11:36:58.588094+0000","last_became_peered":"2024-03-24T11:36:58.588094+0000","last_unstale":"2024-03-24T11:38:51.603204+0000","last_undegraded":"2024-03-24T11:38:51.603204+0000","last_fullsized":"2024-03-24T11:38:51.603204+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'11","last_scrub_stamp":"2024-03-24T11:37:58.943622+0000","last_deep_scrub":"357'10","last_deep_scrub_stamp":"2024-03-24T11:35:53.386928+0000","last_clean_scrub_stamp":"2024-03-24T11:37:58.943622+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:13.122646+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"213'34","reported_seq":846,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607161+0000","last_change":"2024-03-24T11:38:00.140302+0000","last_active":"2024-03-24T11:38:51.607161+0000","last_peered":"2024-03-24T11:38:51.607161+0000","last_clean":"2024-03-24T11:38:51.607161+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:38:51.607161+0000","last_undegraded":"2024-03-24T11:38:51.607161+0000","last_fullsized":"2024-03-24T11:38:51.607161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:38:00.140160+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:38:00.140160+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:10.111079+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":292,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.617321+0000","last_change":"2024-03-24T11:37:37.707970+0000","last_active":"2024-03-24T11:38:51.617321+0000","last_peered":"2024-03-24T11:38:51.617321+0000","last_clean":"2024-03-24T11:38:51.617321+0000","last_became_active":"2024-03-24T11:36:58.595352+0000","last_became_peered":"2024-03-24T11:36:58.595352+0000","last_unstale":"2024-03-24T11:38:51.617321+0000","last_undegraded":"2024-03-24T11:38:51.617321+0000","last_fullsized":"2024-03-24T11:38:51.617321+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:37:37.707835+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.707835+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:52.508470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":788,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607663+0000","last_change":"2024-03-24T11:38:04.034506+0000","last_active":"2024-03-24T11:38:51.607663+0000","last_peered":"2024-03-24T11:38:51.607663+0000","last_clean":"2024-03-24T11:38:51.607663+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:38:51.607663+0000","last_undegraded":"2024-03-24T11:38:51.607663+0000","last_fullsized":"2024-03-24T11:38:51.607663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:04.034411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.034411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:30.306124+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"213'15","reported_seq":781,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.356840+0000","last_change":"2024-03-24T11:35:53.799172+0000","last_active":"2024-03-24T11:38:49.356840+0000","last_peered":"2024-03-24T11:38:49.356840+0000","last_clean":"2024-03-24T11:38:49.356840+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:38:49.356840+0000","last_undegraded":"2024-03-24T11:38:49.356840+0000","last_fullsized":"2024-03-24T11:38:49.356840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:35:53.797708+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:35:53.797708+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105743+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":815,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602661+0000","last_change":"2024-03-24T11:38:00.904773+0000","last_active":"2024-03-24T11:38:51.602661+0000","last_peered":"2024-03-24T11:38:51.602661+0000","last_clean":"2024-03-24T11:38:51.602661+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:38:51.602661+0000","last_undegraded":"2024-03-24T11:38:51.602661+0000","last_fullsized":"2024-03-24T11:38:51.602661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:00.904671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:00.904671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:11.738243+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":783,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608081+0000","last_change":"2024-03-24T11:37:37.114801+0000","last_active":"2024-03-24T11:38:51.608081+0000","last_peered":"2024-03-24T11:38:51.608081+0000","last_clean":"2024-03-24T11:38:51.608081+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:38:51.608081+0000","last_undegraded":"2024-03-24T11:38:51.608081+0000","last_fullsized":"2024-03-24T11:38:51.608081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:37.114698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.114698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:57.697360+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":417,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603667+0000","last_change":"2024-03-24T11:37:54.040251+0000","last_active":"2024-03-24T11:38:51.603667+0000","last_peered":"2024-03-24T11:38:51.603667+0000","last_clean":"2024-03-24T11:38:51.603667+0000","last_became_active":"2024-03-24T11:36:58.574704+0000","last_became_peered":"2024-03-24T11:36:58.574704+0000","last_unstale":"2024-03-24T11:38:51.603667+0000","last_undegraded":"2024-03-24T11:38:51.603667+0000","last_fullsized":"2024-03-24T11:38:51.603667+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:54.040131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.040131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.983328+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"213'19","reported_seq":816,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.620223+0000","last_change":"2024-03-24T11:36:50.776834+0000","last_active":"2024-03-24T11:38:51.620223+0000","last_peered":"2024-03-24T11:38:51.620223+0000","last_clean":"2024-03-24T11:38:51.620223+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:38:51.620223+0000","last_undegraded":"2024-03-24T11:38:51.620223+0000","last_fullsized":"2024-03-24T11:38:51.620223+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:36:50.776723+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.776723+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":86,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602955+0000","last_change":"2024-03-24T11:37:42.909195+0000","last_active":"2024-03-24T11:38:51.602955+0000","last_peered":"2024-03-24T11:38:51.602955+0000","last_clean":"2024-03-24T11:38:51.602955+0000","last_became_active":"2024-03-24T11:36:58.576859+0000","last_became_peered":"2024-03-24T11:36:58.576859+0000","last_unstale":"2024-03-24T11:38:51.602955+0000","last_undegraded":"2024-03-24T11:38:51.602955+0000","last_fullsized":"2024-03-24T11:38:51.602955+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:42.909043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.909043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:09.642694+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":756,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.356762+0000","last_change":"2024-03-24T11:33:32.456675+0000","last_active":"2024-03-24T11:38:49.356762+0000","last_peered":"2024-03-24T11:38:49.356762+0000","last_clean":"2024-03-24T11:38:49.356762+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:38:49.356762+0000","last_undegraded":"2024-03-24T11:38:49.356762+0000","last_fullsized":"2024-03-24T11:38:49.356762+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:32.456539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:33:32.456539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105566+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":801,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602188+0000","last_change":"2024-03-24T11:37:38.501961+0000","last_active":"2024-03-24T11:38:51.602188+0000","last_peered":"2024-03-24T11:38:51.602188+0000","last_clean":"2024-03-24T11:38:51.602188+0000","last_became_active":"2024-03-24T11:36:58.594758+0000","last_became_peered":"2024-03-24T11:36:58.594758+0000","last_unstale":"2024-03-24T11:38:51.602188+0000","last_undegraded":"2024-03-24T11:38:51.602188+0000","last_fullsized":"2024-03-24T11:38:51.602188+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:38.501811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:38.501811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:52.645199+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"213'15","reported_seq":836,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608399+0000","last_change":"2024-03-24T11:38:11.883483+0000","last_active":"2024-03-24T11:38:51.608399+0000","last_peered":"2024-03-24T11:38:51.608399+0000","last_clean":"2024-03-24T11:38:51.608399+0000","last_became_active":"2024-03-24T11:37:33.878054+0000","last_became_peered":"2024-03-24T11:37:33.878054+0000","last_unstale":"2024-03-24T11:38:51.608399+0000","last_undegraded":"2024-03-24T11:38:51.608399+0000","last_fullsized":"2024-03-24T11:38:51.608399+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:11.883396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.883396+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:19.987915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.617430+0000","last_change":"2024-03-24T11:36:55.793667+0000","last_active":"2024-03-24T11:38:51.617430+0000","last_peered":"2024-03-24T11:38:51.617430+0000","last_clean":"2024-03-24T11:38:51.617430+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:38:51.617430+0000","last_undegraded":"2024-03-24T11:38:51.617430+0000","last_fullsized":"2024-03-24T11:38:51.617430+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:55.793540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:55.793540+0000","last_clean_scrub_stamp":"2024-03-24T11:36:55.793540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":1012,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352605+0000","last_change":"2024-03-24T11:36:54.050391+0000","last_active":"2024-03-24T11:38:49.352605+0000","last_peered":"2024-03-24T11:38:49.352605+0000","last_clean":"2024-03-24T11:38:49.352605+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:38:49.352605+0000","last_undegraded":"2024-03-24T11:38:49.352605+0000","last_fullsized":"2024-03-24T11:38:49.352605+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:54.050291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:54.050291+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":92,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.606708+0000","last_change":"2024-03-24T11:38:50.540685+0000","last_active":"2024-03-24T11:38:51.606708+0000","last_peered":"2024-03-24T11:38:51.606708+0000","last_clean":"2024-03-24T11:38:51.606708+0000","last_became_active":"2024-03-24T11:36:58.580967+0000","last_became_peered":"2024-03-24T11:36:58.580967+0000","last_unstale":"2024-03-24T11:38:51.606708+0000","last_undegraded":"2024-03-24T11:38:51.606708+0000","last_fullsized":"2024-03-24T11:38:51.606708+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:50.540148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:38:50.540148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:10.416657+0000","scrub_duration":289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"213'20","reported_seq":790,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352403+0000","last_change":"2024-03-24T11:33:19.230202+0000","last_active":"2024-03-24T11:38:49.352403+0000","last_peered":"2024-03-24T11:38:49.352403+0000","last_clean":"2024-03-24T11:38:49.352403+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:38:49.352403+0000","last_undegraded":"2024-03-24T11:38:49.352403+0000","last_fullsized":"2024-03-24T11:38:49.352403+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:19.229920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.229920+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105517+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":78,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602204+0000","last_change":"2024-03-24T11:36:58.581255+0000","last_active":"2024-03-24T11:38:51.602204+0000","last_peered":"2024-03-24T11:38:51.602204+0000","last_clean":"2024-03-24T11:38:51.602204+0000","last_became_active":"2024-03-24T11:36:58.580909+0000","last_became_peered":"2024-03-24T11:36:58.580909+0000","last_unstale":"2024-03-24T11:38:51.602204+0000","last_undegraded":"2024-03-24T11:38:51.602204+0000","last_fullsized":"2024-03-24T11:38:51.602204+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:53.175665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:36:53.175665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":1039,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608081+0000","last_change":"2024-03-24T11:38:08.961040+0000","last_active":"2024-03-24T11:38:51.608081+0000","last_peered":"2024-03-24T11:38:51.608081+0000","last_clean":"2024-03-24T11:38:51.608081+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:38:51.608081+0000","last_undegraded":"2024-03-24T11:38:51.608081+0000","last_fullsized":"2024-03-24T11:38:51.608081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:38:08.960909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.960909+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.998912+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":771,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.618090+0000","last_change":"2024-03-24T11:36:48.755290+0000","last_active":"2024-03-24T11:38:51.618090+0000","last_peered":"2024-03-24T11:38:51.618090+0000","last_clean":"2024-03-24T11:38:51.618090+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:38:51.618090+0000","last_undegraded":"2024-03-24T11:38:51.618090+0000","last_fullsized":"2024-03-24T11:38:51.618090+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:48.755147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:48.755147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"213'24","reported_seq":475,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603541+0000","last_change":"2024-03-24T11:37:32.834586+0000","last_active":"2024-03-24T11:38:51.603541+0000","last_peered":"2024-03-24T11:38:51.603541+0000","last_clean":"2024-03-24T11:38:51.603541+0000","last_became_active":"2024-03-24T11:36:58.586291+0000","last_became_peered":"2024-03-24T11:36:58.586291+0000","last_unstale":"2024-03-24T11:38:51.603541+0000","last_undegraded":"2024-03-24T11:38:51.603541+0000","last_fullsized":"2024-03-24T11:38:51.603541+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:37:32.834444+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:37:32.834444+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":76,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603273+0000","last_change":"2024-03-24T11:36:58.574750+0000","last_active":"2024-03-24T11:38:51.603273+0000","last_peered":"2024-03-24T11:38:51.603273+0000","last_clean":"2024-03-24T11:38:51.603273+0000","last_became_active":"2024-03-24T11:36:58.574428+0000","last_became_peered":"2024-03-24T11:36:58.574428+0000","last_unstale":"2024-03-24T11:38:51.603273+0000","last_undegraded":"2024-03-24T11:38:51.603273+0000","last_fullsized":"2024-03-24T11:38:51.603273+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:47.206306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:36:47.206306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":799,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602964+0000","last_change":"2024-03-24T11:37:34.760552+0000","last_active":"2024-03-24T11:38:51.602964+0000","last_peered":"2024-03-24T11:38:51.602964+0000","last_clean":"2024-03-24T11:38:51.602964+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:38:51.602964+0000","last_undegraded":"2024-03-24T11:38:51.602964+0000","last_fullsized":"2024-03-24T11:38:51.602964+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:34.760436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:37:34.760436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":760,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.350967+0000","last_change":"2024-03-24T11:37:40.087780+0000","last_active":"2024-03-24T11:38:49.350967+0000","last_peered":"2024-03-24T11:38:49.350967+0000","last_clean":"2024-03-24T11:38:49.350967+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:38:49.350967+0000","last_undegraded":"2024-03-24T11:38:49.350967+0000","last_fullsized":"2024-03-24T11:38:49.350967+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.086223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.086223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.092592+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"213'15","reported_seq":820,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607226+0000","last_change":"2024-03-24T11:37:55.694778+0000","last_active":"2024-03-24T11:38:51.607226+0000","last_peered":"2024-03-24T11:38:51.607226+0000","last_clean":"2024-03-24T11:38:51.607226+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:38:51.607226+0000","last_undegraded":"2024-03-24T11:38:51.607226+0000","last_fullsized":"2024-03-24T11:38:51.607226+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:37:55.694628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.694628+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:03.393716+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":398,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352720+0000","last_change":"2024-03-24T11:37:51.172948+0000","last_active":"2024-03-24T11:38:49.352720+0000","last_peered":"2024-03-24T11:38:49.352720+0000","last_clean":"2024-03-24T11:38:49.352720+0000","last_became_active":"2024-03-24T11:36:58.575459+0000","last_became_peered":"2024-03-24T11:36:58.575459+0000","last_unstale":"2024-03-24T11:38:49.352720+0000","last_undegraded":"2024-03-24T11:38:49.352720+0000","last_fullsized":"2024-03-24T11:38:49.352720+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.172813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.172813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:18.315597+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":756,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352208+0000","last_change":"2024-03-24T11:33:26.388166+0000","last_active":"2024-03-24T11:38:49.352208+0000","last_peered":"2024-03-24T11:38:49.352208+0000","last_clean":"2024-03-24T11:38:49.352208+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:38:49.352208+0000","last_undegraded":"2024-03-24T11:38:49.352208+0000","last_fullsized":"2024-03-24T11:38:49.352208+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:26.387703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:26.387703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":779,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.610490+0000","last_change":"2024-03-24T11:38:50.433428+0000","last_active":"2024-03-24T11:38:51.610490+0000","last_peered":"2024-03-24T11:38:51.610490+0000","last_clean":"2024-03-24T11:38:51.610490+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:38:51.610490+0000","last_undegraded":"2024-03-24T11:38:51.610490+0000","last_fullsized":"2024-03-24T11:38:51.610490+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:50.432934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:50.432934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.785937+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"213'19","reported_seq":791,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.351799+0000","last_change":"2024-03-24T11:33:40.546128+0000","last_active":"2024-03-24T11:38:49.351799+0000","last_peered":"2024-03-24T11:38:49.351799+0000","last_clean":"2024-03-24T11:38:49.351799+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:38:49.351799+0000","last_undegraded":"2024-03-24T11:38:49.351799+0000","last_fullsized":"2024-03-24T11:38:49.351799+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:33:40.545997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:40.545997+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105647+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":799,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.608099+0000","last_change":"2024-03-24T11:38:47.913620+0000","last_active":"2024-03-24T11:38:51.608099+0000","last_peered":"2024-03-24T11:38:51.608099+0000","last_clean":"2024-03-24T11:38:51.608099+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:38:51.608099+0000","last_undegraded":"2024-03-24T11:38:51.608099+0000","last_fullsized":"2024-03-24T11:38:51.608099+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.913497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.913497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:49.332361+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":86,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602642+0000","last_change":"2024-03-24T11:37:01.743777+0000","last_active":"2024-03-24T11:38:51.602642+0000","last_peered":"2024-03-24T11:38:51.602642+0000","last_clean":"2024-03-24T11:38:51.602642+0000","last_became_active":"2024-03-24T11:36:58.577143+0000","last_became_peered":"2024-03-24T11:36:58.577143+0000","last_unstale":"2024-03-24T11:38:51.602642+0000","last_undegraded":"2024-03-24T11:38:51.602642+0000","last_fullsized":"2024-03-24T11:38:51.602642+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.743684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.743684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":760,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352528+0000","last_change":"2024-03-24T11:36:48.160280+0000","last_active":"2024-03-24T11:38:49.352528+0000","last_peered":"2024-03-24T11:38:49.352528+0000","last_clean":"2024-03-24T11:38:49.352528+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:38:49.352528+0000","last_undegraded":"2024-03-24T11:38:49.352528+0000","last_fullsized":"2024-03-24T11:38:49.352528+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:48.160164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:48.160164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105820+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"213'10","reported_seq":774,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.351054+0000","last_change":"2024-03-24T11:33:33.501770+0000","last_active":"2024-03-24T11:38:49.351054+0000","last_peered":"2024-03-24T11:38:49.351054+0000","last_clean":"2024-03-24T11:38:49.351054+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:38:49.351054+0000","last_undegraded":"2024-03-24T11:38:49.351054+0000","last_fullsized":"2024-03-24T11:38:49.351054+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:33:33.501639+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:33:33.501639+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":799,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:53.263999+0000","last_change":"2024-03-24T11:38:53.264039+0000","last_active":"2024-03-24T11:38:53.263999+0000","last_peered":"2024-03-24T11:38:53.263999+0000","last_clean":"2024-03-24T11:38:53.263999+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:38:53.263999+0000","last_undegraded":"2024-03-24T11:38:53.263999+0000","last_fullsized":"2024-03-24T11:38:53.263999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.263761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.263761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:18.212346+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":805,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603305+0000","last_change":"2024-03-24T11:36:58.577856+0000","last_active":"2024-03-24T11:38:51.603305+0000","last_peered":"2024-03-24T11:38:51.603305+0000","last_clean":"2024-03-24T11:38:51.603305+0000","last_became_active":"2024-03-24T11:36:58.577278+0000","last_became_peered":"2024-03-24T11:36:58.577278+0000","last_unstale":"2024-03-24T11:38:51.603305+0000","last_undegraded":"2024-03-24T11:38:51.603305+0000","last_fullsized":"2024-03-24T11:38:51.603305+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_clean_scrub_stamp":"2024-03-24T11:36:56.727031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":81,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.351977+0000","last_change":"2024-03-24T11:38:06.148171+0000","last_active":"2024-03-24T11:38:49.351977+0000","last_peered":"2024-03-24T11:38:49.351977+0000","last_clean":"2024-03-24T11:38:49.351977+0000","last_became_active":"2024-03-24T11:36:58.574827+0000","last_became_peered":"2024-03-24T11:36:58.574827+0000","last_unstale":"2024-03-24T11:38:49.351977+0000","last_undegraded":"2024-03-24T11:38:49.351977+0000","last_fullsized":"2024-03-24T11:38:49.351977+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:06.147813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:06.147813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:35.811713+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"213'24","reported_seq":848,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603737+0000","last_change":"2024-03-24T11:38:11.032460+0000","last_active":"2024-03-24T11:38:51.603737+0000","last_peered":"2024-03-24T11:38:51.603737+0000","last_clean":"2024-03-24T11:38:51.603737+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:38:51.603737+0000","last_undegraded":"2024-03-24T11:38:51.603737+0000","last_fullsized":"2024-03-24T11:38:51.603737+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:38:11.032318+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.032318+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:24.858658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":772,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.350568+0000","last_change":"2024-03-24T11:38:01.211155+0000","last_active":"2024-03-24T11:38:49.350568+0000","last_peered":"2024-03-24T11:38:49.350568+0000","last_clean":"2024-03-24T11:38:49.350568+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:38:49.350568+0000","last_undegraded":"2024-03-24T11:38:49.350568+0000","last_fullsized":"2024-03-24T11:38:49.350568+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.211052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.211052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:14.127032+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":784,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.620310+0000","last_change":"2024-03-24T11:37:59.968770+0000","last_active":"2024-03-24T11:38:51.620310+0000","last_peered":"2024-03-24T11:38:51.620310+0000","last_clean":"2024-03-24T11:38:51.620310+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:38:51.620310+0000","last_undegraded":"2024-03-24T11:38:51.620310+0000","last_fullsized":"2024-03-24T11:38:51.620310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:59.968275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:37:59.968275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:26.500547+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":800,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607282+0000","last_change":"2024-03-24T11:38:12.027553+0000","last_active":"2024-03-24T11:38:51.607282+0000","last_peered":"2024-03-24T11:38:51.607282+0000","last_clean":"2024-03-24T11:38:51.607282+0000","last_became_active":"2024-03-24T11:36:58.592355+0000","last_became_peered":"2024-03-24T11:36:58.592355+0000","last_unstale":"2024-03-24T11:38:51.607282+0000","last_undegraded":"2024-03-24T11:38:51.607282+0000","last_fullsized":"2024-03-24T11:38:51.607282+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.027430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:25.034763+0000","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"213'20","reported_seq":844,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.620048+0000","last_change":"2024-03-24T11:37:36.696164+0000","last_active":"2024-03-24T11:38:51.620048+0000","last_peered":"2024-03-24T11:38:51.620048+0000","last_clean":"2024-03-24T11:38:51.620048+0000","last_became_active":"2024-03-24T11:33:33.699957+0000","last_became_peered":"2024-03-24T11:33:33.699957+0000","last_unstale":"2024-03-24T11:38:51.620048+0000","last_undegraded":"2024-03-24T11:38:51.620048+0000","last_fullsized":"2024-03-24T11:38:51.620048+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:37:36.695829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:36.695829+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:02.287456+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":818,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602631+0000","last_change":"2024-03-24T11:37:52.677447+0000","last_active":"2024-03-24T11:38:51.602631+0000","last_peered":"2024-03-24T11:38:51.602631+0000","last_clean":"2024-03-24T11:38:51.602631+0000","last_became_active":"2024-03-24T11:36:58.593112+0000","last_became_peered":"2024-03-24T11:36:58.593112+0000","last_unstale":"2024-03-24T11:38:51.602631+0000","last_undegraded":"2024-03-24T11:38:51.602631+0000","last_fullsized":"2024-03-24T11:38:51.602631+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:37:52.677294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.677294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:06.104575+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"388'10","reported_seq":1073,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603587+0000","last_change":"2024-03-24T11:38:04.870954+0000","last_active":"2024-03-24T11:38:51.603587+0000","last_peered":"2024-03-24T11:38:51.603587+0000","last_clean":"2024-03-24T11:38:51.603587+0000","last_became_active":"2024-03-24T11:36:58.580932+0000","last_became_peered":"2024-03-24T11:36:58.580932+0000","last_unstale":"2024-03-24T11:38:51.603587+0000","last_undegraded":"2024-03-24T11:38:51.603587+0000","last_fullsized":"2024-03-24T11:38:51.603587+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'10","last_scrub_stamp":"2024-03-24T11:38:04.870822+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.870822+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:15.058342+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":442,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607672+0000","last_change":"2024-03-24T11:37:42.132295+0000","last_active":"2024-03-24T11:38:51.607672+0000","last_peered":"2024-03-24T11:38:51.607672+0000","last_clean":"2024-03-24T11:38:51.607672+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:38:51.607672+0000","last_undegraded":"2024-03-24T11:38:51.607672+0000","last_fullsized":"2024-03-24T11:38:51.607672+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:37:42.132154+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.132154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"213'18","reported_seq":809,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352924+0000","last_change":"2024-03-24T11:36:59.961408+0000","last_active":"2024-03-24T11:38:49.352924+0000","last_peered":"2024-03-24T11:38:49.352924+0000","last_clean":"2024-03-24T11:38:49.352924+0000","last_became_active":"2024-03-24T11:36:58.591714+0000","last_became_peered":"2024-03-24T11:36:58.591714+0000","last_unstale":"2024-03-24T11:38:49.352924+0000","last_undegraded":"2024-03-24T11:38:49.352924+0000","last_fullsized":"2024-03-24T11:38:49.352924+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:36:59.961306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:59.961306+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":84,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602810+0000","last_change":"2024-03-24T11:37:50.620288+0000","last_active":"2024-03-24T11:38:51.602810+0000","last_peered":"2024-03-24T11:38:51.602810+0000","last_clean":"2024-03-24T11:38:51.602810+0000","last_became_active":"2024-03-24T11:36:58.597448+0000","last_became_peered":"2024-03-24T11:36:58.597448+0000","last_unstale":"2024-03-24T11:38:51.602810+0000","last_undegraded":"2024-03-24T11:38:51.602810+0000","last_fullsized":"2024-03-24T11:38:51.602810+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:50.620171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:50.620171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:10.396489+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":83,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352204+0000","last_change":"2024-03-24T11:37:43.066187+0000","last_active":"2024-03-24T11:38:49.352204+0000","last_peered":"2024-03-24T11:38:49.352204+0000","last_clean":"2024-03-24T11:38:49.352204+0000","last_became_active":"2024-03-24T11:36:58.574806+0000","last_became_peered":"2024-03-24T11:36:58.574806+0000","last_unstale":"2024-03-24T11:38:49.352204+0000","last_undegraded":"2024-03-24T11:38:49.352204+0000","last_fullsized":"2024-03-24T11:38:49.352204+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.066063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.066063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:52.830508+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":756,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.356422+0000","last_change":"2024-03-24T11:35:49.842222+0000","last_active":"2024-03-24T11:38:49.356422+0000","last_peered":"2024-03-24T11:38:49.356422+0000","last_clean":"2024-03-24T11:38:49.356422+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:38:49.356422+0000","last_undegraded":"2024-03-24T11:38:49.356422+0000","last_fullsized":"2024-03-24T11:38:49.356422+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:49.842117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:49.842117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105714+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"213'27","reported_seq":853,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.601650+0000","last_change":"2024-03-24T11:37:56.500706+0000","last_active":"2024-03-24T11:38:51.601650+0000","last_peered":"2024-03-24T11:38:51.601650+0000","last_clean":"2024-03-24T11:38:51.601650+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:38:51.601650+0000","last_undegraded":"2024-03-24T11:38:51.601650+0000","last_fullsized":"2024-03-24T11:38:51.601650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:37:56.500311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:56.500311+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.564703+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":796,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.619246+0000","last_change":"2024-03-24T11:36:58.781277+0000","last_active":"2024-03-24T11:38:51.619246+0000","last_peered":"2024-03-24T11:38:51.619246+0000","last_clean":"2024-03-24T11:38:51.619246+0000","last_became_active":"2024-03-24T11:36:58.590975+0000","last_became_peered":"2024-03-24T11:36:58.590975+0000","last_unstale":"2024-03-24T11:38:51.619246+0000","last_undegraded":"2024-03-24T11:38:51.619246+0000","last_fullsized":"2024-03-24T11:38:51.619246+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:58.781161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:36:58.781161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":796,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607158+0000","last_change":"2024-03-24T11:37:43.684412+0000","last_active":"2024-03-24T11:38:51.607158+0000","last_peered":"2024-03-24T11:38:51.607158+0000","last_clean":"2024-03-24T11:38:51.607158+0000","last_became_active":"2024-03-24T11:36:58.596592+0000","last_became_peered":"2024-03-24T11:36:58.596592+0000","last_unstale":"2024-03-24T11:38:51.607158+0000","last_undegraded":"2024-03-24T11:38:51.607158+0000","last_fullsized":"2024-03-24T11:38:51.607158+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.684286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.684286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.427075+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":84,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603570+0000","last_change":"2024-03-24T11:37:48.869860+0000","last_active":"2024-03-24T11:38:51.603570+0000","last_peered":"2024-03-24T11:38:51.603570+0000","last_clean":"2024-03-24T11:38:51.603570+0000","last_became_active":"2024-03-24T11:36:58.575356+0000","last_became_peered":"2024-03-24T11:36:58.575356+0000","last_unstale":"2024-03-24T11:38:51.603570+0000","last_undegraded":"2024-03-24T11:38:51.603570+0000","last_fullsized":"2024-03-24T11:38:51.603570+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:48.869684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:37:48.869684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:56.054301+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"213'7","reported_seq":436,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352005+0000","last_change":"2024-03-24T11:36:58.602264+0000","last_active":"2024-03-24T11:38:49.352005+0000","last_peered":"2024-03-24T11:38:49.352005+0000","last_clean":"2024-03-24T11:38:49.352005+0000","last_became_active":"2024-03-24T11:36:58.586221+0000","last_became_peered":"2024-03-24T11:36:58.586221+0000","last_unstale":"2024-03-24T11:38:49.352005+0000","last_undegraded":"2024-03-24T11:38:49.352005+0000","last_fullsized":"2024-03-24T11:38:49.352005+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:36:04.265719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:04.265719+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":796,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607808+0000","last_change":"2024-03-24T11:37:41.153332+0000","last_active":"2024-03-24T11:38:51.607808+0000","last_peered":"2024-03-24T11:38:51.607808+0000","last_clean":"2024-03-24T11:38:51.607808+0000","last_became_active":"2024-03-24T11:36:58.585301+0000","last_became_peered":"2024-03-24T11:36:58.585301+0000","last_unstale":"2024-03-24T11:38:51.607808+0000","last_undegraded":"2024-03-24T11:38:51.607808+0000","last_fullsized":"2024-03-24T11:38:51.607808+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.153221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.153221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:00.046302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":86,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602640+0000","last_change":"2024-03-24T11:37:01.675006+0000","last_active":"2024-03-24T11:38:51.602640+0000","last_peered":"2024-03-24T11:38:51.602640+0000","last_clean":"2024-03-24T11:38:51.602640+0000","last_became_active":"2024-03-24T11:36:58.598728+0000","last_became_peered":"2024-03-24T11:36:58.598728+0000","last_unstale":"2024-03-24T11:38:51.602640+0000","last_undegraded":"2024-03-24T11:38:51.602640+0000","last_fullsized":"2024-03-24T11:38:51.602640+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.674916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.674916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":779,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607383+0000","last_change":"2024-03-24T11:37:46.149252+0000","last_active":"2024-03-24T11:38:51.607383+0000","last_peered":"2024-03-24T11:38:51.607383+0000","last_clean":"2024-03-24T11:38:51.607383+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:38:51.607383+0000","last_undegraded":"2024-03-24T11:38:51.607383+0000","last_fullsized":"2024-03-24T11:38:51.607383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:46.149176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:37:46.149176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:02.169889+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"213'15","reported_seq":444,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.357131+0000","last_change":"2024-03-24T11:36:58.606754+0000","last_active":"2024-03-24T11:38:49.357131+0000","last_peered":"2024-03-24T11:38:49.357131+0000","last_clean":"2024-03-24T11:38:49.357131+0000","last_became_active":"2024-03-24T11:36:58.580893+0000","last_became_peered":"2024-03-24T11:36:58.580893+0000","last_unstale":"2024-03-24T11:38:49.357131+0000","last_undegraded":"2024-03-24T11:38:49.357131+0000","last_fullsized":"2024-03-24T11:38:49.357131+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:33:24.845898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:24.845898+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105359+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":796,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607963+0000","last_change":"2024-03-24T11:36:58.590875+0000","last_active":"2024-03-24T11:38:51.607963+0000","last_peered":"2024-03-24T11:38:51.607963+0000","last_clean":"2024-03-24T11:38:51.607963+0000","last_became_active":"2024-03-24T11:36:58.590154+0000","last_became_peered":"2024-03-24T11:36:58.590154+0000","last_unstale":"2024-03-24T11:38:51.607963+0000","last_undegraded":"2024-03-24T11:38:51.607963+0000","last_fullsized":"2024-03-24T11:38:51.607963+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:52.302009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:36:52.302009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":796,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607421+0000","last_change":"2024-03-24T11:37:54.088530+0000","last_active":"2024-03-24T11:38:51.607421+0000","last_peered":"2024-03-24T11:38:51.607421+0000","last_clean":"2024-03-24T11:38:51.607421+0000","last_became_active":"2024-03-24T11:37:33.876071+0000","last_became_peered":"2024-03-24T11:37:33.876071+0000","last_unstale":"2024-03-24T11:38:51.607421+0000","last_undegraded":"2024-03-24T11:38:51.607421+0000","last_fullsized":"2024-03-24T11:38:51.607421+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:54.088400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.088400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:13.266896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":779,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.610863+0000","last_change":"2024-03-24T11:37:31.701772+0000","last_active":"2024-03-24T11:38:51.610863+0000","last_peered":"2024-03-24T11:38:51.610863+0000","last_clean":"2024-03-24T11:38:51.610863+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:38:51.610863+0000","last_undegraded":"2024-03-24T11:38:51.610863+0000","last_fullsized":"2024-03-24T11:38:51.610863+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:31.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:31.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"213'27","reported_seq":803,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.351906+0000","last_change":"2024-03-24T11:37:02.002116+0000","last_active":"2024-03-24T11:38:49.351906+0000","last_peered":"2024-03-24T11:38:49.351906+0000","last_clean":"2024-03-24T11:38:49.351906+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:38:49.351906+0000","last_undegraded":"2024-03-24T11:38:49.351906+0000","last_fullsized":"2024-03-24T11:38:49.351906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:37:02.002019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:02.002019+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106206+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":765,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.353218+0000","last_change":"2024-03-24T11:35:51.761976+0000","last_active":"2024-03-24T11:38:49.353218+0000","last_peered":"2024-03-24T11:38:49.353218+0000","last_clean":"2024-03-24T11:38:49.353218+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:38:49.353218+0000","last_undegraded":"2024-03-24T11:38:49.353218+0000","last_fullsized":"2024-03-24T11:38:49.353218+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:51.761850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:51.761850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.105796+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":85,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602873+0000","last_change":"2024-03-24T11:37:00.746342+0000","last_active":"2024-03-24T11:38:51.602873+0000","last_peered":"2024-03-24T11:38:51.602873+0000","last_clean":"2024-03-24T11:38:51.602873+0000","last_became_active":"2024-03-24T11:36:58.584540+0000","last_became_peered":"2024-03-24T11:36:58.584540+0000","last_unstale":"2024-03-24T11:38:51.602873+0000","last_undegraded":"2024-03-24T11:38:51.602873+0000","last_fullsized":"2024-03-24T11:38:51.602873+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:00.746264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:00.746264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":751,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-03-24T11:38:49.352296+0000","last_change":"2024-03-24T11:36:53.164614+0000","last_active":"2024-03-24T11:38:49.352296+0000","last_peered":"2024-03-24T11:38:49.352296+0000","last_clean":"2024-03-24T11:38:49.352296+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:38:49.352296+0000","last_undegraded":"2024-03-24T11:38:49.352296+0000","last_fullsized":"2024-03-24T11:38:49.352296+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:53.164514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:53.164514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:51.106120+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"213'15","reported_seq":478,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.603071+0000","last_change":"2024-03-24T11:37:54.965402+0000","last_active":"2024-03-24T11:38:51.603071+0000","last_peered":"2024-03-24T11:38:51.603071+0000","last_clean":"2024-03-24T11:38:51.603071+0000","last_became_active":"2024-03-24T11:36:58.580796+0000","last_became_peered":"2024-03-24T11:36:58.580796+0000","last_unstale":"2024-03-24T11:38:51.603071+0000","last_undegraded":"2024-03-24T11:38:51.603071+0000","last_fullsized":"2024-03-24T11:38:51.603071+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:37:54.965309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.965309+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:55.562802+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":792,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.617112+0000","last_change":"2024-03-24T11:38:08.039863+0000","last_active":"2024-03-24T11:38:51.617112+0000","last_peered":"2024-03-24T11:38:51.617112+0000","last_clean":"2024-03-24T11:38:51.617112+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:38:51.617112+0000","last_undegraded":"2024-03-24T11:38:51.617112+0000","last_fullsized":"2024-03-24T11:38:51.617112+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:08.039401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.039401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:21.805864+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.602723+0000","last_change":"2024-03-24T11:37:40.752135+0000","last_active":"2024-03-24T11:38:51.602723+0000","last_peered":"2024-03-24T11:38:51.602723+0000","last_clean":"2024-03-24T11:38:51.602723+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:38:51.602723+0000","last_undegraded":"2024-03-24T11:38:51.602723+0000","last_fullsized":"2024-03-24T11:38:51.602723+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.752016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.752016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":84,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-03-24T11:38:51.607358+0000","last_change":"2024-03-24T11:37:51.062312+0000","last_active":"2024-03-24T11:38:51.607358+0000","last_peered":"2024-03-24T11:38:51.607358+0000","last_clean":"2024-03-24T11:38:51.607358+0000","last_became_active":"2024-03-24T11:36:58.589661+0000","last_became_peered":"2024-03-24T11:36:58.589661+0000","last_unstale":"2024-03-24T11:38:51.607358+0000","last_undegraded":"2024-03-24T11:38:51.607358+0000","last_fullsized":"2024-03-24T11:38:51.607358+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.062184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.062184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:57.346693+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6427246616,"num_objects":98073,"num_object_clones":0,"num_object_copies":294219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98073,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142519,"num_write_kb":7698881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19169030144,"data_stored":19169017928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70419,"ondisk_log_size":70419,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":503,"num_read_kb":498,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":2492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":739,"ondisk_log_size":739,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070708,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3438060,"kb_used_data":2721056,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":90307092,"statfs":{"total":95995035648,"available":92474462208,"internally_reserved":0,"allocated":2786361344,"data_stored":2785908873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":734198219},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365049,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3805488,"kb_used_data":3134248,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":89939664,"statfs":{"total":95995035648,"available":92098215936,"internally_reserved":0,"allocated":3209469952,"data_stored":3209009974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":687274438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575123,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4390728,"kb_used_data":3524800,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":89354424,"statfs":{"total":95995035648,"available":91498930176,"internally_reserved":0,"allocated":3609395200,"data_stored":3608910979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738648,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3492016,"kb_used_data":2725032,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":90253136,"statfs":{"total":95995035648,"available":92419211264,"internally_reserved":0,"allocated":2790432768,"data_stored":2789971968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411598,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2996716,"kb_used_data":2333860,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":90748436,"statfs":{"total":95995035648,"available":92926398464,"internally_reserved":0,"allocated":2389872640,"data_stored":2389416221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149958,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3641184,"kb_used_data":2851992,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":90103968,"statfs":{"total":95995035648,"available":92266463232,"internally_reserved":0,"allocated":2920439808,"data_stored":2919980222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":808122823},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659390,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2338356,"kb_used_data":1558888,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":91406796,"statfs":{"total":95995035648,"available":93600559104,"internally_reserved":0,"allocated":1596301312,"data_stored":1595841287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2779054080,"data_stored":2779054080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3577679872,"data_stored":3577675800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2378698752,"data_stored":2378694680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2882207744,"data_stored":2882207744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1583415296,"data_stored":1583415296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3189837824,"data_stored":3189833752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2778136576,"data_stored":2778136576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:39:01.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:39:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:01.346 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:01.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:01.447 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:01.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:01.493 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:01.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:01.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:01.623 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:01.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:01.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:01.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:01.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:01.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T11:39:01.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:01 smithi181 ceph-mon[33998]: pgmap v925: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T11:39:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:01 smithi181 ceph-mon[33998]: 11.5 scrub starts 2024-03-24T11:39:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:01 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:39:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:01 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:39:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:01 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:39:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:01 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:39:01.926 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:01.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[29608]: pgmap v925: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T11:39:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[29608]: 11.5 scrub starts 2024-03-24T11:39:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:39:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[37907]: pgmap v925: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[37907]: 11.5 scrub starts 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:39:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:01 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:39:02.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:02.079 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:02.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:02.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:02.242 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:02.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:02.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:02.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:02.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:02.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:02.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:02.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:02.546 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:02.598 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:02.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:02.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:02.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:02.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:02.786 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:02.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 11.5 scrub ok 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 11.6 scrub ok 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:39:02.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:39:02.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 11.5 scrub ok 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 11.6 scrub ok 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:39:02.797 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:02 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:39:02.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 11.5 scrub ok 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 11.6 scrub ok 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:39:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:02 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:39:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:02.927 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:02.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:03.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:03.055 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:03.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:03.157 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:03.197 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:03.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:03.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:03.338 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:03.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:03.368 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:03.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:03.474 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:03.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:03.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:03.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:03.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:03.676 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:39:03.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:03.740 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:03.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:03.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:03.882 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:39:03.882 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: pgmap v926: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3098727176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: 11.4 scrub starts 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: pgmap v926: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3098727176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:03.883 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:03.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: 11.4 scrub starts 2024-03-24T11:39:03.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:39:03.884 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:03 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:39:03.887 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:03.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:03.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:04.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:04.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: pgmap v926: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3098727176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: 11.4 scrub starts 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:39:04.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:03 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:39:04.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.161 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:04.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:04.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:04.293 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:04.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.435 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:04.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:04.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:04.567 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:04.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:39:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:39:04.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:04.694 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:04.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:04.854 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:39:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:39:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[29608]: osdmap e422: 8 total, 8 up, 7 in 2024-03-24T11:39:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:04.923 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:04.924 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[29608]: osdmap e423: 8 total, 8 up, 7 in 2024-03-24T11:39:04.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:39:04.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:39:04.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[37907]: osdmap e422: 8 total, 8 up, 7 in 2024-03-24T11:39:04.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:04.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:04.925 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:04 smithi179 ceph-mon[37907]: osdmap e423: 8 total, 8 up, 7 in 2024-03-24T11:39:04.957 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:04.999 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:05.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:05.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:05.146 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:05.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:05.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:04 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:39:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:04 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:39:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:04 smithi181 ceph-mon[33998]: osdmap e422: 8 total, 8 up, 7 in 2024-03-24T11:39:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:04 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:04 smithi181 ceph-mon[33998]: osdmap e423: 8 total, 8 up, 7 in 2024-03-24T11:39:05.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:39:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:39:05.247 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:05.292 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:05.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:05.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:05.421 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:05.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:05.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:05.559 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:05.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:05.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:05.691 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:05.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:05.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:05.822 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:05.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:05.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:05.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:05.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:05.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:05.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[37907]: 11.4 scrub ok 2024-03-24T11:39:05.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[37907]: pgmap v929: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[29608]: 11.4 scrub ok 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[29608]: pgmap v929: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:39:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:39:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:39:05.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:05 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:39:06.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:06.080 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:06.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:06.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:06.154 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:05 smithi181 ceph-mon[33998]: 11.4 scrub ok 2024-03-24T11:39:06.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:05 smithi181 ceph-mon[33998]: pgmap v929: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T11:39:06.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:05 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:39:06.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:05 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:39:06.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:05 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:39:06.155 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:05 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:39:06.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:06.225 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:06.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:06.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:06.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:06.350 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:06.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:06.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:06.476 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:06.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:06.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:06.621 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:06.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:06.722 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:06.746 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:06.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:06.755 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[29608]: 5.10 deep-scrub starts 2024-03-24T11:39:06.755 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[37907]: 5.10 deep-scrub starts 2024-03-24T11:39:06.848 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:06.887 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:06.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:06.989 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.017 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:07.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[29608]: 5.10 deep-scrub ok 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[37907]: 5.10 deep-scrub ok 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:39:07.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:39:07.084 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:39:07.084 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:06 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:39:07.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.142 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:07.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:07.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:06 smithi181 ceph-mon[33998]: 5.10 deep-scrub starts 2024-03-24T11:39:07.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:06 smithi181 ceph-mon[33998]: 5.10 deep-scrub ok 2024-03-24T11:39:07.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:06 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:39:07.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:06 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:39:07.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:06 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:39:07.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:06 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:39:07.244 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:07.292 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:07.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.418 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:07.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:07.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.556 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:07.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:07.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:39:07.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.706 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:07.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:07.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:07.837 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:07.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[37907]: pgmap v930: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:39:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[37907]: from='client.15912 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:39:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:39:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:39:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[37907]: 3.12 scrub starts 2024-03-24T11:39:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[37907]: 3.12 scrub ok 2024-03-24T11:39:07.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[29608]: pgmap v930: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:39:07.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[29608]: from='client.15912 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:39:07.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:39:07.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:39:07.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[29608]: 3.12 scrub starts 2024-03-24T11:39:07.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:07 smithi179 ceph-mon[29608]: 3.12 scrub ok 2024-03-24T11:39:07.971 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:07.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:08.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:08.109 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:08.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:08.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:07 smithi181 ceph-mon[33998]: pgmap v930: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 53 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-24T11:39:08.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:07 smithi181 ceph-mon[33998]: from='client.15912 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:39:08.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:07 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:39:08.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:07 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:39:08.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:07 smithi181 ceph-mon[33998]: 3.12 scrub starts 2024-03-24T11:39:08.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:07 smithi181 ceph-mon[33998]: 3.12 scrub ok 2024-03-24T11:39:08.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:08.245 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:08.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:08.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:08.370 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:08.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:08.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:08.518 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:08.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:08.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:08.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:08.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:08.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:08.792 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:08.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:08.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:08.925 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:08.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:08 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:39:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:08 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:39:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:08 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:39:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:08 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:39:09.026 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:09.071 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:09.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:08 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:39:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:08 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:39:09.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:09.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:09.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:09.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:09.326 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:09.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:09.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:09.468 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:09.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:09.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:09.607 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:09.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:09.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:09.731 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:09.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:09.833 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:09.871 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:09.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:09.900 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:39:09.910 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:39:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:09 smithi179 ceph-mon[29608]: pgmap v931: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 946 op/s 2024-03-24T11:39:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:09 smithi179 ceph-mon[29608]: 11.f scrub starts 2024-03-24T11:39:09.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:09 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:39:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:09 smithi179 ceph-mon[37907]: pgmap v931: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 946 op/s 2024-03-24T11:39:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:09 smithi179 ceph-mon[37907]: 11.f scrub starts 2024-03-24T11:39:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:09 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:39:09.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:10.016 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:10.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:10.116 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:09 smithi181 ceph-mon[33998]: pgmap v931: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 946 op/s 2024-03-24T11:39:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:09 smithi181 ceph-mon[33998]: 11.f scrub starts 2024-03-24T11:39:10.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:09 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:39:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:10.167 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:10.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:10.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:10.300 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:10.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:10.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:10.429 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:10.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:10.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:10.607 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:10.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:10.708 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:10.737 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:10.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:10.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:10.892 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:10.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:10 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:39:10.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:10 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:10.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:10 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:39:10.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:10 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:10.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:11.067 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:11.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:11.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:10 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:39:11.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:10 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:39:11.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:11.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:11.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:11.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:11.328 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:11.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:11.430 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:11.479 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:11.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:11.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:11.606 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:11.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:11.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:11.751 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:11.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:11.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:11.880 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:11.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:11.890 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[29608]: 11.f scrub ok 2024-03-24T11:39:11.890 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[29608]: pgmap v932: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-03-24T11:39:11.890 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:39:11.890 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:39:11.891 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[37907]: 11.f scrub ok 2024-03-24T11:39:11.891 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[37907]: pgmap v932: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-03-24T11:39:11.891 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:39:11.891 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:11 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:39:11.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:12.017 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:12.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:12.149 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:12.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:11 smithi181 ceph-mon[33998]: 11.f scrub ok 2024-03-24T11:39:12.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:11 smithi181 ceph-mon[33998]: pgmap v932: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-03-24T11:39:12.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:11 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:39:12.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:11 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:39:12.245 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":930,"stamp":"2024-03-24T11:39:06.454941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5608446326,"num_objects":85768,"num_object_clones":0,"num_object_copies":257304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85768,"num_whiteouts":0,"num_read":2078,"num_read_kb":1830,"num_write":156026,"num_write_kb":7698926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":331231,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71720,"ondisk_log_size":71720,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":21986120,"kb_used_data":16643468,"kb_used_omap":10,"kb_used_meta":5342389,"kb_avail":634229944,"statfs":{"total":671965249536,"available":649451462656,"internally_reserved":0,"allocated":17042911232,"data_stored":17039638062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":5470606444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,8,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-630849536,"num_objects":-9626,"num_object_clones":0,"num_object_copies":-28878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9626,"num_whiteouts":0,"num_read":428,"num_read_kb":428,"num_write":9908,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004303"},"pg_stats":[{"pgid":"11.3","version":"421'9715","reported_seq":9869,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.608871+0000","last_change":"2024-03-24T11:38:58.631183+0000","last_active":"2024-03-24T11:39:03.608871+0000","last_peered":"2024-03-24T11:39:03.608871+0000","last_clean":"2024-03-24T11:39:03.608871+0000","last_became_active":"2024-03-24T11:36:51.539490+0000","last_became_peered":"2024-03-24T11:36:51.539490+0000","last_unstale":"2024-03-24T11:39:03.608871+0000","last_undegraded":"2024-03-24T11:39:03.608871+0000","last_fullsized":"2024-03-24T11:39:03.608871+0000","mapping_epoch":382,"log_start":"407'5100","ondisk_log_start":"407'5100","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9353","last_scrub_stamp":"2024-03-24T11:38:58.630837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:58.630837+0000","objects_scrubbed":5739,"log_size":4615,"log_dups_size":0,"ondisk_log_size":4615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.602366+0000","scrub_duration":1593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348323840,"num_objects":5315,"num_object_clones":0,"num_object_copies":15945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9715,"num_write_kb":480960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"421'30","reported_seq":482,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.360185+0000","last_change":"2024-03-24T11:38:10.116849+0000","last_active":"2024-03-24T11:39:04.360185+0000","last_peered":"2024-03-24T11:39:04.360185+0000","last_clean":"2024-03-24T11:39:04.360185+0000","last_became_active":"2024-03-24T11:36:58.591623+0000","last_became_peered":"2024-03-24T11:36:58.591623+0000","last_unstale":"2024-03-24T11:39:04.360185+0000","last_undegraded":"2024-03-24T11:39:04.360185+0000","last_fullsized":"2024-03-24T11:39:04.360185+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:10.116699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.116699+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:20.360374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":778,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.361309+0000","last_change":"2024-03-24T11:36:58.586904+0000","last_active":"2024-03-24T11:39:04.361309+0000","last_peered":"2024-03-24T11:39:04.361309+0000","last_clean":"2024-03-24T11:39:04.361309+0000","last_became_active":"2024-03-24T11:36:58.586602+0000","last_became_peered":"2024-03-24T11:36:58.586602+0000","last_unstale":"2024-03-24T11:39:04.361309+0000","last_undegraded":"2024-03-24T11:39:04.361309+0000","last_fullsized":"2024-03-24T11:39:04.361309+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:25.353054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:25.353054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"362'9","reported_seq":1062,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.672826+0000","last_change":"2024-03-24T11:37:37.413749+0000","last_active":"2024-03-24T11:39:03.672826+0000","last_peered":"2024-03-24T11:39:03.672826+0000","last_clean":"2024-03-24T11:39:03.672826+0000","last_became_active":"2024-03-24T11:18:57.701267+0000","last_became_peered":"2024-03-24T11:18:57.701267+0000","last_unstale":"2024-03-24T11:39:03.672826+0000","last_undegraded":"2024-03-24T11:39:03.672826+0000","last_fullsized":"2024-03-24T11:39:03.672826+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'9","last_scrub_stamp":"2024-03-24T11:37:37.413561+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.413561+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:06.050090+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":86,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.374851+0000","last_change":"2024-03-24T11:38:01.792439+0000","last_active":"2024-03-24T11:38:54.374851+0000","last_peered":"2024-03-24T11:38:54.374851+0000","last_clean":"2024-03-24T11:38:54.374851+0000","last_became_active":"2024-03-24T11:36:58.590765+0000","last_became_peered":"2024-03-24T11:36:58.590765+0000","last_unstale":"2024-03-24T11:38:54.374851+0000","last_undegraded":"2024-03-24T11:38:54.374851+0000","last_fullsized":"2024-03-24T11:38:54.374851+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.792132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.792132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:24.724156+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":856,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.357793+0000","last_change":"2024-03-24T11:36:57.973025+0000","last_active":"2024-03-24T11:39:04.357793+0000","last_peered":"2024-03-24T11:39:04.357793+0000","last_clean":"2024-03-24T11:39:04.357793+0000","last_became_active":"2024-03-24T11:30:35.359701+0000","last_became_peered":"2024-03-24T11:30:35.359701+0000","last_unstale":"2024-03-24T11:39:04.357793+0000","last_undegraded":"2024-03-24T11:39:04.357793+0000","last_fullsized":"2024-03-24T11:39:04.357793+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:57.972898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:57.972898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"423'9730","reported_seq":9822,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:05.226853+0000","last_change":"2024-03-24T11:36:51.538337+0000","last_active":"2024-03-24T11:39:05.226853+0000","last_peered":"2024-03-24T11:39:05.226853+0000","last_clean":"2024-03-24T11:39:05.226853+0000","last_became_active":"2024-03-24T11:36:51.537990+0000","last_became_peered":"2024-03-24T11:36:51.537990+0000","last_unstale":"2024-03-24T11:39:05.226853+0000","last_undegraded":"2024-03-24T11:39:05.226853+0000","last_fullsized":"2024-03-24T11:39:05.226853+0000","mapping_epoch":382,"log_start":"409'5600","ondisk_log_start":"409'5600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345505792,"num_objects":5272,"num_object_clones":0,"num_object_copies":15816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":480064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"421'53","reported_seq":559,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.672707+0000","last_change":"2024-03-24T11:38:04.488873+0000","last_active":"2024-03-24T11:39:03.672707+0000","last_peered":"2024-03-24T11:39:03.672707+0000","last_clean":"2024-03-24T11:39:03.672707+0000","last_became_active":"2024-03-24T11:36:58.580651+0000","last_became_peered":"2024-03-24T11:36:58.580651+0000","last_unstale":"2024-03-24T11:39:03.672707+0000","last_undegraded":"2024-03-24T11:39:03.672707+0000","last_fullsized":"2024-03-24T11:39:03.672707+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:38:04.488661+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.488661+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:30.545546+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":835,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368354+0000","last_change":"2024-03-24T11:38:48.018756+0000","last_active":"2024-03-24T11:38:54.368354+0000","last_peered":"2024-03-24T11:38:54.368354+0000","last_clean":"2024-03-24T11:38:54.368354+0000","last_became_active":"2024-03-24T11:36:58.587097+0000","last_became_peered":"2024-03-24T11:36:58.587097+0000","last_unstale":"2024-03-24T11:38:54.368354+0000","last_undegraded":"2024-03-24T11:38:54.368354+0000","last_fullsized":"2024-03-24T11:38:54.368354+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:38:48.018617+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.018617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.449456+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":799,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.367933+0000","last_change":"2024-03-24T11:37:04.724933+0000","last_active":"2024-03-24T11:38:54.367933+0000","last_peered":"2024-03-24T11:38:54.367933+0000","last_clean":"2024-03-24T11:38:54.367933+0000","last_became_active":"2024-03-24T11:36:58.595572+0000","last_became_peered":"2024-03-24T11:36:58.595572+0000","last_unstale":"2024-03-24T11:38:54.367933+0000","last_undegraded":"2024-03-24T11:38:54.367933+0000","last_fullsized":"2024-03-24T11:38:54.367933+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:04.724780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:37:04.724780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":115,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.364087+0000","last_change":"2024-03-24T11:38:52.569860+0000","last_active":"2024-03-24T11:39:04.364087+0000","last_peered":"2024-03-24T11:39:04.364087+0000","last_clean":"2024-03-24T11:39:04.364087+0000","last_became_active":"2024-03-24T11:36:58.582225+0000","last_became_peered":"2024-03-24T11:36:58.582225+0000","last_unstale":"2024-03-24T11:39:04.364087+0000","last_undegraded":"2024-03-24T11:39:04.364087+0000","last_fullsized":"2024-03-24T11:39:04.364087+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:52.569730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.569730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.998514+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":958,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369795+0000","last_change":"2024-03-24T11:38:03.034263+0000","last_active":"2024-03-24T11:38:54.369795+0000","last_peered":"2024-03-24T11:38:54.369795+0000","last_clean":"2024-03-24T11:38:54.369795+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:38:54.369795+0000","last_undegraded":"2024-03-24T11:38:54.369795+0000","last_fullsized":"2024-03-24T11:38:54.369795+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:38:03.034131+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:38:03.034131+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:20.878170+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"421'9658","reported_seq":9819,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.608935+0000","last_change":"2024-03-24T11:38:48.786761+0000","last_active":"2024-03-24T11:39:03.608935+0000","last_peered":"2024-03-24T11:39:03.608935+0000","last_clean":"2024-03-24T11:39:03.608935+0000","last_became_active":"2024-03-24T11:36:51.529788+0000","last_became_peered":"2024-03-24T11:36:51.529788+0000","last_unstale":"2024-03-24T11:39:03.608935+0000","last_undegraded":"2024-03-24T11:39:03.608935+0000","last_fullsized":"2024-03-24T11:39:03.608935+0000","mapping_epoch":382,"log_start":"407'5100","ondisk_log_start":"407'5100","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'8678","last_scrub_stamp":"2024-03-24T11:38:48.786544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.786544+0000","objects_scrubbed":6471,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:10.325229+0000","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356122624,"num_objects":5434,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9658,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":484,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.404806+0000","last_change":"2024-03-24T11:37:31.892734+0000","last_active":"2024-03-24T11:39:03.404806+0000","last_peered":"2024-03-24T11:39:03.404806+0000","last_clean":"2024-03-24T11:39:03.404806+0000","last_became_active":"2024-03-24T11:36:58.587116+0000","last_became_peered":"2024-03-24T11:36:58.587116+0000","last_unstale":"2024-03-24T11:39:03.404806+0000","last_undegraded":"2024-03-24T11:39:03.404806+0000","last_fullsized":"2024-03-24T11:39:03.404806+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:37:31.892595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:31.892595+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":88,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.670715+0000","last_change":"2024-03-24T11:38:47.247564+0000","last_active":"2024-03-24T11:39:03.670715+0000","last_peered":"2024-03-24T11:39:03.670715+0000","last_clean":"2024-03-24T11:39:03.670715+0000","last_became_active":"2024-03-24T11:36:58.585229+0000","last_became_peered":"2024-03-24T11:36:58.585229+0000","last_unstale":"2024-03-24T11:39:03.670715+0000","last_undegraded":"2024-03-24T11:39:03.670715+0000","last_fullsized":"2024-03-24T11:39:03.670715+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.247390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.247390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:17.162245+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":791,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370608+0000","last_change":"2024-03-24T11:37:38.108397+0000","last_active":"2024-03-24T11:38:54.370608+0000","last_peered":"2024-03-24T11:38:54.370608+0000","last_clean":"2024-03-24T11:38:54.370608+0000","last_became_active":"2024-03-24T11:36:58.575297+0000","last_became_peered":"2024-03-24T11:36:58.575297+0000","last_unstale":"2024-03-24T11:38:54.370608+0000","last_undegraded":"2024-03-24T11:38:54.370608+0000","last_fullsized":"2024-03-24T11:38:54.370608+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:38.108325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:37:38.108325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:04.811029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":760,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.359938+0000","last_change":"2024-03-24T11:33:38.472649+0000","last_active":"2024-03-24T11:39:04.359938+0000","last_peered":"2024-03-24T11:39:04.359938+0000","last_clean":"2024-03-24T11:39:04.359938+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:39:04.359938+0000","last_undegraded":"2024-03-24T11:39:04.359938+0000","last_fullsized":"2024-03-24T11:39:04.359938+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:38.472418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.472418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"421'9577","reported_seq":9666,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.408163+0000","last_change":"2024-03-24T11:36:51.545822+0000","last_active":"2024-03-24T11:39:03.408163+0000","last_peered":"2024-03-24T11:39:03.408163+0000","last_clean":"2024-03-24T11:39:03.408163+0000","last_became_active":"2024-03-24T11:36:51.545544+0000","last_became_peered":"2024-03-24T11:36:51.545544+0000","last_unstale":"2024-03-24T11:39:03.408163+0000","last_undegraded":"2024-03-24T11:39:03.408163+0000","last_fullsized":"2024-03-24T11:39:03.408163+0000","mapping_epoch":382,"log_start":"408'5400","ondisk_log_start":"408'5400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4177,"log_dups_size":0,"ondisk_log_size":4177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353828864,"num_objects":5399,"num_object_clones":0,"num_object_copies":16197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9577,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"421'42","reported_seq":881,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.050958+0000","last_change":"2024-03-24T11:36:58.618200+0000","last_active":"2024-03-24T11:39:03.050958+0000","last_peered":"2024-03-24T11:39:03.050958+0000","last_clean":"2024-03-24T11:39:03.050958+0000","last_became_active":"2024-03-24T11:36:58.594679+0000","last_became_peered":"2024-03-24T11:36:58.594679+0000","last_unstale":"2024-03-24T11:39:03.050958+0000","last_undegraded":"2024-03-24T11:39:03.050958+0000","last_fullsized":"2024-03-24T11:39:03.050958+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:36:56.684283+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:36:56.684283+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":47,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":781,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.357420+0000","last_change":"2024-03-24T11:37:01.005665+0000","last_active":"2024-03-24T11:39:04.357420+0000","last_peered":"2024-03-24T11:39:04.357420+0000","last_clean":"2024-03-24T11:39:04.357420+0000","last_became_active":"2024-03-24T11:18:53.682039+0000","last_became_peered":"2024-03-24T11:18:53.682039+0000","last_unstale":"2024-03-24T11:39:04.357420+0000","last_undegraded":"2024-03-24T11:39:04.357420+0000","last_fullsized":"2024-03-24T11:39:04.357420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.005586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.005586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'9","reported_seq":148,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370121+0000","last_change":"2024-03-24T11:38:51.903763+0000","last_active":"2024-03-24T11:38:54.370121+0000","last_peered":"2024-03-24T11:38:54.370121+0000","last_clean":"2024-03-24T11:38:54.370121+0000","last_became_active":"2024-03-24T11:36:58.574227+0000","last_became_peered":"2024-03-24T11:36:58.574227+0000","last_unstale":"2024-03-24T11:38:54.370121+0000","last_undegraded":"2024-03-24T11:38:54.370121+0000","last_fullsized":"2024-03-24T11:38:54.370121+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-03-24T11:38:51.903639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:51.903639+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.344849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":781,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:55.092380+0000","last_change":"2024-03-24T11:38:55.092421+0000","last_active":"2024-03-24T11:38:55.092380+0000","last_peered":"2024-03-24T11:38:55.092380+0000","last_clean":"2024-03-24T11:38:55.092380+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:38:55.092380+0000","last_undegraded":"2024-03-24T11:38:55.092380+0000","last_fullsized":"2024-03-24T11:38:55.092380+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:55.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:55.092309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.593383+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":949,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368211+0000","last_change":"2024-03-24T11:37:44.069186+0000","last_active":"2024-03-24T11:38:54.368211+0000","last_peered":"2024-03-24T11:38:54.368211+0000","last_clean":"2024-03-24T11:38:54.368211+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:38:54.368211+0000","last_undegraded":"2024-03-24T11:38:54.368211+0000","last_fullsized":"2024-03-24T11:38:54.368211+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:44.069073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.069073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:07.819002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"421'9626","reported_seq":9712,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.426621+0000","last_change":"2024-03-24T11:36:51.545109+0000","last_active":"2024-03-24T11:39:03.426621+0000","last_peered":"2024-03-24T11:39:03.426621+0000","last_clean":"2024-03-24T11:39:03.426621+0000","last_became_active":"2024-03-24T11:36:51.544798+0000","last_became_peered":"2024-03-24T11:36:51.544798+0000","last_unstale":"2024-03-24T11:39:03.426621+0000","last_undegraded":"2024-03-24T11:39:03.426621+0000","last_fullsized":"2024-03-24T11:39:03.426621+0000","mapping_epoch":382,"log_start":"408'5400","ondisk_log_start":"408'5400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348913664,"num_objects":5324,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9626,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":92,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.359754+0000","last_change":"2024-03-24T11:37:41.083113+0000","last_active":"2024-03-24T11:39:04.359754+0000","last_peered":"2024-03-24T11:39:04.359754+0000","last_clean":"2024-03-24T11:39:04.359754+0000","last_became_active":"2024-03-24T11:36:58.579874+0000","last_became_peered":"2024-03-24T11:36:58.579874+0000","last_unstale":"2024-03-24T11:39:04.359754+0000","last_undegraded":"2024-03-24T11:39:04.359754+0000","last_fullsized":"2024-03-24T11:39:04.359754+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.082984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.082984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"421'42","reported_seq":838,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.357495+0000","last_change":"2024-03-24T11:37:32.916715+0000","last_active":"2024-03-24T11:39:04.357495+0000","last_peered":"2024-03-24T11:39:04.357495+0000","last_clean":"2024-03-24T11:39:04.357495+0000","last_became_active":"2024-03-24T11:18:56.141205+0000","last_became_peered":"2024-03-24T11:18:56.141205+0000","last_unstale":"2024-03-24T11:39:04.357495+0000","last_undegraded":"2024-03-24T11:39:04.357495+0000","last_fullsized":"2024-03-24T11:39:04.357495+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:37:32.916579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:32.916579+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":781,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.363743+0000","last_change":"2024-03-24T11:38:57.213449+0000","last_active":"2024-03-24T11:39:04.363743+0000","last_peered":"2024-03-24T11:39:04.363743+0000","last_clean":"2024-03-24T11:39:04.363743+0000","last_became_active":"2024-03-24T11:18:53.678854+0000","last_became_peered":"2024-03-24T11:18:53.678854+0000","last_unstale":"2024-03-24T11:39:04.363743+0000","last_undegraded":"2024-03-24T11:39:04.363743+0000","last_fullsized":"2024-03-24T11:39:04.363743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:57.213332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.213332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:19.264365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":83,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.362330+0000","last_change":"2024-03-24T11:36:58.575117+0000","last_active":"2024-03-24T11:39:04.362330+0000","last_peered":"2024-03-24T11:39:04.362330+0000","last_clean":"2024-03-24T11:39:04.362330+0000","last_became_active":"2024-03-24T11:36:58.574742+0000","last_became_peered":"2024-03-24T11:36:58.574742+0000","last_unstale":"2024-03-24T11:39:04.362330+0000","last_undegraded":"2024-03-24T11:39:04.362330+0000","last_fullsized":"2024-03-24T11:39:04.362330+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:30.982770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:33:30.982770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":404,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.672921+0000","last_change":"2024-03-24T11:38:54.329326+0000","last_active":"2024-03-24T11:39:03.672921+0000","last_peered":"2024-03-24T11:39:03.672921+0000","last_clean":"2024-03-24T11:39:03.672921+0000","last_became_active":"2024-03-24T11:37:01.307977+0000","last_became_peered":"2024-03-24T11:37:01.307977+0000","last_unstale":"2024-03-24T11:39:03.672921+0000","last_undegraded":"2024-03-24T11:39:03.672921+0000","last_fullsized":"2024-03-24T11:39:03.672921+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:38:54.329204+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_clean_scrub_stamp":"2024-03-24T11:38:54.329204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:20.416911+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"421'9514","reported_seq":9661,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.387137+0000","last_change":"2024-03-24T11:38:57.081858+0000","last_active":"2024-03-24T11:39:03.387137+0000","last_peered":"2024-03-24T11:39:03.387137+0000","last_clean":"2024-03-24T11:39:03.387137+0000","last_became_active":"2024-03-24T11:36:58.595829+0000","last_became_peered":"2024-03-24T11:36:58.595829+0000","last_unstale":"2024-03-24T11:39:03.387137+0000","last_undegraded":"2024-03-24T11:39:03.387137+0000","last_fullsized":"2024-03-24T11:39:03.387137+0000","mapping_epoch":387,"log_start":"408'5300","ondisk_log_start":"408'5300","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9080","last_scrub_stamp":"2024-03-24T11:38:57.081681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.081681+0000","objects_scrubbed":5915,"log_size":4214,"log_dups_size":0,"ondisk_log_size":4214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:59.502930+0000","scrub_duration":2532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354156544,"num_objects":5404,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9514,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"388'9","reported_seq":1027,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.357615+0000","last_change":"2024-03-24T11:36:58.931284+0000","last_active":"2024-03-24T11:39:04.357615+0000","last_peered":"2024-03-24T11:39:04.357615+0000","last_clean":"2024-03-24T11:39:04.357615+0000","last_became_active":"2024-03-24T11:36:58.591546+0000","last_became_peered":"2024-03-24T11:36:58.591546+0000","last_unstale":"2024-03-24T11:39:04.357615+0000","last_undegraded":"2024-03-24T11:39:04.357615+0000","last_fullsized":"2024-03-24T11:39:04.357615+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:58.931185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:58.931185+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'42","reported_seq":501,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.438036+0000","last_change":"2024-03-24T11:37:35.115468+0000","last_active":"2024-03-24T11:39:03.438036+0000","last_peered":"2024-03-24T11:39:03.438036+0000","last_clean":"2024-03-24T11:39:03.438036+0000","last_became_active":"2024-03-24T11:36:58.573987+0000","last_became_peered":"2024-03-24T11:36:58.573987+0000","last_unstale":"2024-03-24T11:39:03.438036+0000","last_undegraded":"2024-03-24T11:39:03.438036+0000","last_fullsized":"2024-03-24T11:39:03.438036+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'28","last_scrub_stamp":"2024-03-24T11:37:35.115336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.115336+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":813,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.371033+0000","last_change":"2024-03-24T11:38:09.054727+0000","last_active":"2024-03-24T11:38:54.371033+0000","last_peered":"2024-03-24T11:38:54.371033+0000","last_clean":"2024-03-24T11:38:54.371033+0000","last_became_active":"2024-03-24T11:36:58.594655+0000","last_became_peered":"2024-03-24T11:36:58.594655+0000","last_unstale":"2024-03-24T11:38:54.371033+0000","last_undegraded":"2024-03-24T11:38:54.371033+0000","last_fullsized":"2024-03-24T11:38:54.371033+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:09.054428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:09.054428+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:29.739709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":414,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369569+0000","last_change":"2024-03-24T11:38:10.119555+0000","last_active":"2024-03-24T11:38:54.369569+0000","last_peered":"2024-03-24T11:38:54.369569+0000","last_clean":"2024-03-24T11:38:54.369569+0000","last_became_active":"2024-03-24T11:30:35.356661+0000","last_became_peered":"2024-03-24T11:30:35.356661+0000","last_unstale":"2024-03-24T11:38:54.369569+0000","last_undegraded":"2024-03-24T11:38:54.369569+0000","last_fullsized":"2024-03-24T11:38:54.369569+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:10.119478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.119478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:19.763125+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":970,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.359810+0000","last_change":"2024-03-24T11:37:42.045811+0000","last_active":"2024-03-24T11:39:04.359810+0000","last_peered":"2024-03-24T11:39:04.359810+0000","last_clean":"2024-03-24T11:39:04.359810+0000","last_became_active":"2024-03-24T11:35:52.362283+0000","last_became_peered":"2024-03-24T11:35:52.362283+0000","last_unstale":"2024-03-24T11:39:04.359810+0000","last_undegraded":"2024-03-24T11:39:04.359810+0000","last_fullsized":"2024-03-24T11:39:04.359810+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:42.045652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.045652+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"422'9831","reported_seq":9999,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.696010+0000","last_change":"2024-03-24T11:38:53.481972+0000","last_active":"2024-03-24T11:39:03.696010+0000","last_peered":"2024-03-24T11:39:03.696010+0000","last_clean":"2024-03-24T11:39:03.696010+0000","last_became_active":"2024-03-24T11:36:58.589952+0000","last_became_peered":"2024-03-24T11:36:58.589952+0000","last_unstale":"2024-03-24T11:39:03.696010+0000","last_undegraded":"2024-03-24T11:39:03.696010+0000","last_fullsized":"2024-03-24T11:39:03.696010+0000","mapping_epoch":387,"log_start":"404'4700","ondisk_log_start":"404'4700","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'9128","last_scrub_stamp":"2024-03-24T11:38:53.481477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.481477+0000","objects_scrubbed":6260,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.975145+0000","scrub_duration":2214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359923736,"num_objects":5493,"num_object_clones":0,"num_object_copies":16479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9831,"num_write_kb":490305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":1045,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:02.353189+0000","last_change":"2024-03-24T11:39:02.353239+0000","last_active":"2024-03-24T11:39:02.353189+0000","last_peered":"2024-03-24T11:39:02.353189+0000","last_clean":"2024-03-24T11:39:02.353189+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:39:02.353189+0000","last_undegraded":"2024-03-24T11:39:02.353189+0000","last_fullsized":"2024-03-24T11:39:02.353189+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:39:02.353110+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.353110+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.696468+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'30","reported_seq":869,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.300328+0000","last_change":"2024-03-24T11:38:53.578501+0000","last_active":"2024-03-24T11:39:03.300328+0000","last_peered":"2024-03-24T11:39:03.300328+0000","last_clean":"2024-03-24T11:39:03.300328+0000","last_became_active":"2024-03-24T11:36:58.597641+0000","last_became_peered":"2024-03-24T11:36:58.597641+0000","last_unstale":"2024-03-24T11:39:03.300328+0000","last_undegraded":"2024-03-24T11:39:03.300328+0000","last_fullsized":"2024-03-24T11:39:03.300328+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:53.578365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.578365+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:19.628891+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":805,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370075+0000","last_change":"2024-03-24T11:37:45.089273+0000","last_active":"2024-03-24T11:38:54.370075+0000","last_peered":"2024-03-24T11:38:54.370075+0000","last_clean":"2024-03-24T11:38:54.370075+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:38:54.370075+0000","last_undegraded":"2024-03-24T11:38:54.370075+0000","last_fullsized":"2024-03-24T11:38:54.370075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:45.089152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_clean_scrub_stamp":"2024-03-24T11:37:45.089152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":783,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:01.453611+0000","last_change":"2024-03-24T11:39:01.453653+0000","last_active":"2024-03-24T11:39:01.453611+0000","last_peered":"2024-03-24T11:39:01.453611+0000","last_clean":"2024-03-24T11:39:01.453611+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:39:01.453611+0000","last_undegraded":"2024-03-24T11:39:01.453611+0000","last_fullsized":"2024-03-24T11:39:01.453611+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:39:01.453536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.453536+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.228643+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":986,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369097+0000","last_change":"2024-03-24T11:37:41.909224+0000","last_active":"2024-03-24T11:38:54.369097+0000","last_peered":"2024-03-24T11:38:54.369097+0000","last_clean":"2024-03-24T11:38:54.369097+0000","last_became_active":"2024-03-24T11:35:39.758022+0000","last_became_peered":"2024-03-24T11:35:39.758022+0000","last_unstale":"2024-03-24T11:38:54.369097+0000","last_undegraded":"2024-03-24T11:38:54.369097+0000","last_fullsized":"2024-03-24T11:38:54.369097+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:41.909070+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.909070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.644204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"421'9530","reported_seq":9624,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.427481+0000","last_change":"2024-03-24T11:36:58.590495+0000","last_active":"2024-03-24T11:39:03.427481+0000","last_peered":"2024-03-24T11:39:03.427481+0000","last_clean":"2024-03-24T11:39:03.427481+0000","last_became_active":"2024-03-24T11:36:58.590179+0000","last_became_peered":"2024-03-24T11:36:58.590179+0000","last_unstale":"2024-03-24T11:39:03.427481+0000","last_undegraded":"2024-03-24T11:39:03.427481+0000","last_fullsized":"2024-03-24T11:39:03.427481+0000","mapping_epoch":387,"log_start":"408'5300","ondisk_log_start":"408'5300","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354418688,"num_objects":5408,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9530,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":810,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368632+0000","last_change":"2024-03-24T11:37:43.909192+0000","last_active":"2024-03-24T11:38:54.368632+0000","last_peered":"2024-03-24T11:38:54.368632+0000","last_clean":"2024-03-24T11:38:54.368632+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:38:54.368632+0000","last_undegraded":"2024-03-24T11:38:54.368632+0000","last_fullsized":"2024-03-24T11:38:54.368632+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.909062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.909062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:01.175526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'39","reported_seq":856,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.257560+0000","last_change":"2024-03-24T11:38:04.942863+0000","last_active":"2024-03-24T11:39:03.257560+0000","last_peered":"2024-03-24T11:39:03.257560+0000","last_clean":"2024-03-24T11:39:03.257560+0000","last_became_active":"2024-03-24T11:18:56.141687+0000","last_became_peered":"2024-03-24T11:18:56.141687+0000","last_unstale":"2024-03-24T11:39:03.257560+0000","last_undegraded":"2024-03-24T11:39:03.257560+0000","last_fullsized":"2024-03-24T11:39:03.257560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:38:04.942715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.942715+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:34.510965+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":810,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.367670+0000","last_change":"2024-03-24T11:37:49.656241+0000","last_active":"2024-03-24T11:38:54.367670+0000","last_peered":"2024-03-24T11:38:54.367670+0000","last_clean":"2024-03-24T11:38:54.367670+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:38:54.367670+0000","last_undegraded":"2024-03-24T11:38:54.367670+0000","last_fullsized":"2024-03-24T11:38:54.367670+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:49.656104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:49.656104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.626409+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":790,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.366543+0000","last_change":"2024-03-24T11:37:45.747922+0000","last_active":"2024-03-24T11:38:54.366543+0000","last_peered":"2024-03-24T11:38:54.366543+0000","last_clean":"2024-03-24T11:38:54.366543+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:38:54.366543+0000","last_undegraded":"2024-03-24T11:38:54.366543+0000","last_fullsized":"2024-03-24T11:38:54.366543+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:45.747799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:45.747799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:58.909782+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":986,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368591+0000","last_change":"2024-03-24T11:37:46.878302+0000","last_active":"2024-03-24T11:38:54.368591+0000","last_peered":"2024-03-24T11:38:54.368591+0000","last_clean":"2024-03-24T11:38:54.368591+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:38:54.368591+0000","last_undegraded":"2024-03-24T11:38:54.368591+0000","last_fullsized":"2024-03-24T11:38:54.368591+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:46.878216+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:33:32.751865+0000","last_clean_scrub_stamp":"2024-03-24T11:37:46.878216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"423'9895","reported_seq":9987,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:05.219257+0000","last_change":"2024-03-24T11:36:51.535382+0000","last_active":"2024-03-24T11:39:05.219257+0000","last_peered":"2024-03-24T11:39:05.219257+0000","last_clean":"2024-03-24T11:39:05.219257+0000","last_became_active":"2024-03-24T11:36:51.534255+0000","last_became_peered":"2024-03-24T11:36:51.534255+0000","last_unstale":"2024-03-24T11:39:05.219257+0000","last_undegraded":"2024-03-24T11:39:05.219257+0000","last_fullsized":"2024-03-24T11:39:05.219257+0000","mapping_epoch":382,"log_start":"409'5700","ondisk_log_start":"409'5700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349110272,"num_objects":5327,"num_object_clones":0,"num_object_copies":15981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":786,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370613+0000","last_change":"2024-03-24T11:37:43.142179+0000","last_active":"2024-03-24T11:38:54.370613+0000","last_peered":"2024-03-24T11:38:54.370613+0000","last_clean":"2024-03-24T11:38:54.370613+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:38:54.370613+0000","last_undegraded":"2024-03-24T11:38:54.370613+0000","last_fullsized":"2024-03-24T11:38:54.370613+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.142063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.142063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'36","reported_seq":361,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:02.946527+0000","last_change":"2024-03-24T11:39:01.923217+0000","last_active":"2024-03-24T11:39:02.946527+0000","last_peered":"2024-03-24T11:39:02.946527+0000","last_clean":"2024-03-24T11:39:02.946527+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:39:02.946527+0000","last_undegraded":"2024-03-24T11:39:02.946527+0000","last_fullsized":"2024-03-24T11:39:02.946527+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'28","last_scrub_stamp":"2024-03-24T11:39:01.923083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.923083+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:20.126246+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":956,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:02.280869+0000","last_change":"2024-03-24T11:37:03.767302+0000","last_active":"2024-03-24T11:39:02.280869+0000","last_peered":"2024-03-24T11:39:02.280869+0000","last_clean":"2024-03-24T11:39:02.280869+0000","last_became_active":"2024-03-24T11:30:35.360881+0000","last_became_peered":"2024-03-24T11:30:35.360881+0000","last_unstale":"2024-03-24T11:39:02.280869+0000","last_undegraded":"2024-03-24T11:39:02.280869+0000","last_fullsized":"2024-03-24T11:39:02.280869+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:37:03.767151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:37:03.767151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":508,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":803,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.361560+0000","last_change":"2024-03-24T11:36:58.573150+0000","last_active":"2024-03-24T11:38:54.361560+0000","last_peered":"2024-03-24T11:38:54.361560+0000","last_clean":"2024-03-24T11:38:54.361560+0000","last_became_active":"2024-03-24T11:36:58.572807+0000","last_became_peered":"2024-03-24T11:36:58.572807+0000","last_unstale":"2024-03-24T11:38:54.361560+0000","last_undegraded":"2024-03-24T11:38:54.361560+0000","last_fullsized":"2024-03-24T11:38:54.361560+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:56.781495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:36:56.781495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":798,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.367362+0000","last_change":"2024-03-24T11:38:05.656269+0000","last_active":"2024-03-24T11:38:54.367362+0000","last_peered":"2024-03-24T11:38:54.367362+0000","last_clean":"2024-03-24T11:38:54.367362+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:38:54.367362+0000","last_undegraded":"2024-03-24T11:38:54.367362+0000","last_fullsized":"2024-03-24T11:38:54.367362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.656146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.656146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:28.915290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":998,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.374248+0000","last_change":"2024-03-24T11:37:01.308329+0000","last_active":"2024-03-24T11:38:54.374248+0000","last_peered":"2024-03-24T11:38:54.374248+0000","last_clean":"2024-03-24T11:38:54.374248+0000","last_became_active":"2024-03-24T11:37:01.307925+0000","last_became_peered":"2024-03-24T11:37:01.307925+0000","last_unstale":"2024-03-24T11:38:54.374248+0000","last_undegraded":"2024-03-24T11:38:54.374248+0000","last_fullsized":"2024-03-24T11:38:54.374248+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:49.592471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:49.592471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"423'9666","reported_seq":9755,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:05.233828+0000","last_change":"2024-03-24T11:36:51.533861+0000","last_active":"2024-03-24T11:39:05.233828+0000","last_peered":"2024-03-24T11:39:05.233828+0000","last_clean":"2024-03-24T11:39:05.233828+0000","last_became_active":"2024-03-24T11:36:51.533003+0000","last_became_peered":"2024-03-24T11:36:51.533003+0000","last_unstale":"2024-03-24T11:39:05.233828+0000","last_undegraded":"2024-03-24T11:39:05.233828+0000","last_fullsized":"2024-03-24T11:39:05.233828+0000","mapping_epoch":382,"log_start":"408'5500","ondisk_log_start":"408'5500","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345899008,"num_objects":5278,"num_object_clones":0,"num_object_copies":15834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":416,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.680326+0000","last_change":"2024-03-24T11:38:05.438477+0000","last_active":"2024-03-24T11:39:03.680326+0000","last_peered":"2024-03-24T11:39:03.680326+0000","last_clean":"2024-03-24T11:39:03.680326+0000","last_became_active":"2024-03-24T11:30:35.355674+0000","last_became_peered":"2024-03-24T11:30:35.355674+0000","last_unstale":"2024-03-24T11:39:03.680326+0000","last_undegraded":"2024-03-24T11:39:03.680326+0000","last_fullsized":"2024-03-24T11:39:03.680326+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.438380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.438380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:33.739925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"421'49","reported_seq":1146,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.356124+0000","last_change":"2024-03-24T11:37:52.944624+0000","last_active":"2024-03-24T11:39:03.356124+0000","last_peered":"2024-03-24T11:39:03.356124+0000","last_clean":"2024-03-24T11:39:03.356124+0000","last_became_active":"2024-03-24T11:36:58.574117+0000","last_became_peered":"2024-03-24T11:36:58.574117+0000","last_unstale":"2024-03-24T11:39:03.356124+0000","last_undegraded":"2024-03-24T11:39:03.356124+0000","last_fullsized":"2024-03-24T11:39:03.356124+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-03-24T11:37:52.944533+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.944533+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:16.899416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":787,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.362009+0000","last_change":"2024-03-24T11:37:35.979186+0000","last_active":"2024-03-24T11:39:04.362009+0000","last_peered":"2024-03-24T11:39:04.362009+0000","last_clean":"2024-03-24T11:39:04.362009+0000","last_became_active":"2024-03-24T11:27:12.329549+0000","last_became_peered":"2024-03-24T11:27:12.329549+0000","last_unstale":"2024-03-24T11:39:04.362009+0000","last_undegraded":"2024-03-24T11:39:04.362009+0000","last_fullsized":"2024-03-24T11:39:04.362009+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.978638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.978638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":742,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.394334+0000","last_change":"2024-03-24T11:39:03.394379+0000","last_active":"2024-03-24T11:39:03.394334+0000","last_peered":"2024-03-24T11:39:03.394334+0000","last_clean":"2024-03-24T11:39:03.394334+0000","last_became_active":"2024-03-24T11:35:35.730443+0000","last_became_peered":"2024-03-24T11:35:35.730443+0000","last_unstale":"2024-03-24T11:39:03.394334+0000","last_undegraded":"2024-03-24T11:39:03.394334+0000","last_fullsized":"2024-03-24T11:39:03.394334+0000","mapping_epoch":237,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:39:03.394244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:39:03.394244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:04.309803+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":804,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370384+0000","last_change":"2024-03-24T11:38:02.291361+0000","last_active":"2024-03-24T11:38:54.370384+0000","last_peered":"2024-03-24T11:38:54.370384+0000","last_clean":"2024-03-24T11:38:54.370384+0000","last_became_active":"2024-03-24T11:36:58.586913+0000","last_became_peered":"2024-03-24T11:36:58.586913+0000","last_unstale":"2024-03-24T11:38:54.370384+0000","last_undegraded":"2024-03-24T11:38:54.370384+0000","last_fullsized":"2024-03-24T11:38:54.370384+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:02.291240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:38:02.291240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.313586+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"423'9788","reported_seq":9880,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:05.237274+0000","last_change":"2024-03-24T11:36:51.530980+0000","last_active":"2024-03-24T11:39:05.237274+0000","last_peered":"2024-03-24T11:39:05.237274+0000","last_clean":"2024-03-24T11:39:05.237274+0000","last_became_active":"2024-03-24T11:36:51.530664+0000","last_became_peered":"2024-03-24T11:36:51.530664+0000","last_unstale":"2024-03-24T11:39:05.237274+0000","last_undegraded":"2024-03-24T11:39:05.237274+0000","last_fullsized":"2024-03-24T11:39:05.237274+0000","mapping_epoch":382,"log_start":"408'5600","ondisk_log_start":"408'5600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345505792,"num_objects":5272,"num_object_clones":0,"num_object_copies":15816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9788,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":812,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.673349+0000","last_change":"2024-03-24T11:37:57.458128+0000","last_active":"2024-03-24T11:39:03.673349+0000","last_peered":"2024-03-24T11:39:03.673349+0000","last_clean":"2024-03-24T11:39:03.673349+0000","last_became_active":"2024-03-24T11:18:57.702268+0000","last_became_peered":"2024-03-24T11:18:57.702268+0000","last_unstale":"2024-03-24T11:39:03.673349+0000","last_undegraded":"2024-03-24T11:39:03.673349+0000","last_fullsized":"2024-03-24T11:39:03.673349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:57.457058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:57.457058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:11.932433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"421'21","reported_seq":830,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.673776+0000","last_change":"2024-03-24T11:39:01.291942+0000","last_active":"2024-03-24T11:39:03.673776+0000","last_peered":"2024-03-24T11:39:03.673776+0000","last_clean":"2024-03-24T11:39:03.673776+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:39:03.673776+0000","last_undegraded":"2024-03-24T11:39:03.673776+0000","last_fullsized":"2024-03-24T11:39:03.673776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'19","last_scrub_stamp":"2024-03-24T11:39:01.291795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.291795+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.103347+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":829,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.670763+0000","last_change":"2024-03-24T11:37:59.474830+0000","last_active":"2024-03-24T11:39:03.670763+0000","last_peered":"2024-03-24T11:39:03.670763+0000","last_clean":"2024-03-24T11:39:03.670763+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:39:03.670763+0000","last_undegraded":"2024-03-24T11:39:03.670763+0000","last_fullsized":"2024-03-24T11:39:03.670763+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:59.474714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:37:59.474714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:25.516288+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":811,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.365130+0000","last_change":"2024-03-24T11:37:53.683206+0000","last_active":"2024-03-24T11:38:54.365130+0000","last_peered":"2024-03-24T11:38:54.365130+0000","last_clean":"2024-03-24T11:38:54.365130+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:38:54.365130+0000","last_undegraded":"2024-03-24T11:38:54.365130+0000","last_fullsized":"2024-03-24T11:38:54.365130+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:53.683090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:53.683090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:10.088022+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":414,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:55.095212+0000","last_change":"2024-03-24T11:38:55.095212+0000","last_active":"2024-03-24T11:38:55.095212+0000","last_peered":"2024-03-24T11:38:55.095212+0000","last_clean":"2024-03-24T11:38:55.095212+0000","last_became_active":"2024-03-24T11:30:35.352714+0000","last_became_peered":"2024-03-24T11:30:35.352714+0000","last_unstale":"2024-03-24T11:38:55.095212+0000","last_undegraded":"2024-03-24T11:38:55.095212+0000","last_fullsized":"2024-03-24T11:38:55.095212+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:55.094992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:38:55.094992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:22.520992+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":757,"reported_epoch":421,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-24T11:38:59.395234+0000","last_change":"2024-03-24T11:38:59.395234+0000","last_active":"2024-03-24T11:38:59.395234+0000","last_peered":"2024-03-24T11:38:59.395234+0000","last_clean":"2024-03-24T11:36:55.645795+0000","last_became_active":"2024-03-24T11:36:59.373164+0000","last_became_peered":"2024-03-24T11:36:59.373164+0000","last_unstale":"2024-03-24T11:38:59.395234+0000","last_undegraded":"2024-03-24T11:38:59.395234+0000","last_fullsized":"2024-03-24T11:38:59.395234+0000","mapping_epoch":388,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:00.701398+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.701398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,4],"avail_no_missing":["0","2","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"423'9846","reported_seq":9937,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:05.223947+0000","last_change":"2024-03-24T11:36:51.532092+0000","last_active":"2024-03-24T11:39:05.223947+0000","last_peered":"2024-03-24T11:39:05.223947+0000","last_clean":"2024-03-24T11:39:05.223947+0000","last_became_active":"2024-03-24T11:36:51.531785+0000","last_became_peered":"2024-03-24T11:36:51.531785+0000","last_unstale":"2024-03-24T11:39:05.223947+0000","last_undegraded":"2024-03-24T11:39:05.223947+0000","last_fullsized":"2024-03-24T11:39:05.223947+0000","mapping_epoch":382,"log_start":"409'5700","ondisk_log_start":"409'5700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344064000,"num_objects":5250,"num_object_clones":0,"num_object_copies":15750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":483072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":792,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.372281+0000","last_change":"2024-03-24T11:37:51.788499+0000","last_active":"2024-03-24T11:38:54.372281+0000","last_peered":"2024-03-24T11:38:54.372281+0000","last_clean":"2024-03-24T11:38:54.372281+0000","last_became_active":"2024-03-24T11:36:58.594154+0000","last_became_peered":"2024-03-24T11:36:58.594154+0000","last_unstale":"2024-03-24T11:38:54.372281+0000","last_undegraded":"2024-03-24T11:38:54.372281+0000","last_fullsized":"2024-03-24T11:38:54.372281+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.788360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.788360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:56.964510+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"421'53","reported_seq":1152,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.330626+0000","last_change":"2024-03-24T11:37:44.707841+0000","last_active":"2024-03-24T11:39:03.330626+0000","last_peered":"2024-03-24T11:39:03.330626+0000","last_clean":"2024-03-24T11:39:03.330626+0000","last_became_active":"2024-03-24T11:37:33.888447+0000","last_became_peered":"2024-03-24T11:37:33.888447+0000","last_unstale":"2024-03-24T11:39:03.330626+0000","last_undegraded":"2024-03-24T11:39:03.330626+0000","last_fullsized":"2024-03-24T11:39:03.330626+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'41","last_scrub_stamp":"2024-03-24T11:37:44.707693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.707693+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":87,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.360746+0000","last_change":"2024-03-24T11:38:01.964312+0000","last_active":"2024-03-24T11:38:54.360746+0000","last_peered":"2024-03-24T11:38:54.360746+0000","last_clean":"2024-03-24T11:38:54.360746+0000","last_became_active":"2024-03-24T11:36:58.591467+0000","last_became_peered":"2024-03-24T11:36:58.591467+0000","last_unstale":"2024-03-24T11:38:54.360746+0000","last_undegraded":"2024-03-24T11:38:54.360746+0000","last_fullsized":"2024-03-24T11:38:54.360746+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.964189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.348439+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":805,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368627+0000","last_change":"2024-03-24T11:37:03.354928+0000","last_active":"2024-03-24T11:38:54.368627+0000","last_peered":"2024-03-24T11:38:54.368627+0000","last_clean":"2024-03-24T11:38:54.368627+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:38:54.368627+0000","last_undegraded":"2024-03-24T11:38:54.368627+0000","last_fullsized":"2024-03-24T11:38:54.368627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:37:03.354797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:37:03.354797+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":775,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.364749+0000","last_change":"2024-03-24T11:39:02.217365+0000","last_active":"2024-03-24T11:39:04.364749+0000","last_peered":"2024-03-24T11:39:04.364749+0000","last_clean":"2024-03-24T11:39:04.364749+0000","last_became_active":"2024-03-24T11:36:58.578703+0000","last_became_peered":"2024-03-24T11:36:58.578703+0000","last_unstale":"2024-03-24T11:39:04.364749+0000","last_undegraded":"2024-03-24T11:39:04.364749+0000","last_fullsized":"2024-03-24T11:39:04.364749+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:02.217239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.217239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.525641+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":980,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368211+0000","last_change":"2024-03-24T11:37:55.105164+0000","last_active":"2024-03-24T11:38:54.368211+0000","last_peered":"2024-03-24T11:38:54.368211+0000","last_clean":"2024-03-24T11:38:54.368211+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:38:54.368211+0000","last_undegraded":"2024-03-24T11:38:54.368211+0000","last_fullsized":"2024-03-24T11:38:54.368211+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:55.105007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.105007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.410544+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"421'9598","reported_seq":9711,"reported_epoch":421,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T11:39:03.611459+0000","last_change":"2024-03-24T11:39:03.076909+0000","last_active":"2024-03-24T11:39:03.611459+0000","last_peered":"2024-03-24T11:39:03.611459+0000","last_clean":"2024-03-24T11:39:03.611459+0000","last_became_active":"2024-03-24T11:36:51.539297+0000","last_became_peered":"2024-03-24T11:36:51.539297+0000","last_unstale":"2024-03-24T11:39:03.611459+0000","last_undegraded":"2024-03-24T11:39:03.611459+0000","last_fullsized":"2024-03-24T11:39:03.611459+0000","mapping_epoch":382,"log_start":"407'5000","ondisk_log_start":"407'5000","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":2024,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347209728,"num_objects":5298,"num_object_clones":0,"num_object_copies":15894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9598,"num_write_kb":476672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":839,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.240177+0000","last_change":"2024-03-24T11:36:58.617099+0000","last_active":"2024-03-24T11:39:03.240177+0000","last_peered":"2024-03-24T11:39:03.240177+0000","last_clean":"2024-03-24T11:39:03.240177+0000","last_became_active":"2024-03-24T11:36:58.590879+0000","last_became_peered":"2024-03-24T11:36:58.590879+0000","last_unstale":"2024-03-24T11:39:03.240177+0000","last_undegraded":"2024-03-24T11:39:03.240177+0000","last_fullsized":"2024-03-24T11:39:03.240177+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:36:54.778072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:54.778072+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":807,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369181+0000","last_change":"2024-03-24T11:38:05.022910+0000","last_active":"2024-03-24T11:38:54.369181+0000","last_peered":"2024-03-24T11:38:54.369181+0000","last_clean":"2024-03-24T11:38:54.369181+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:38:54.369181+0000","last_undegraded":"2024-03-24T11:38:54.369181+0000","last_fullsized":"2024-03-24T11:38:54.369181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:05.022785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.022785+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:20.167706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":92,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.362085+0000","last_change":"2024-03-24T11:39:01.186035+0000","last_active":"2024-03-24T11:39:04.362085+0000","last_peered":"2024-03-24T11:39:04.362085+0000","last_clean":"2024-03-24T11:39:04.362085+0000","last_became_active":"2024-03-24T11:36:58.576200+0000","last_became_peered":"2024-03-24T11:36:58.576200+0000","last_unstale":"2024-03-24T11:39:04.362085+0000","last_undegraded":"2024-03-24T11:39:04.362085+0000","last_fullsized":"2024-03-24T11:39:04.362085+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:01.185721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.185721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:27.043011+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":782,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.361867+0000","last_change":"2024-03-24T11:37:35.857267+0000","last_active":"2024-03-24T11:38:54.361867+0000","last_peered":"2024-03-24T11:38:54.361867+0000","last_clean":"2024-03-24T11:38:54.361867+0000","last_became_active":"2024-03-24T11:18:59.714171+0000","last_became_peered":"2024-03-24T11:18:59.714171+0000","last_unstale":"2024-03-24T11:38:54.361867+0000","last_undegraded":"2024-03-24T11:38:54.361867+0000","last_fullsized":"2024-03-24T11:38:54.361867+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.856841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.856841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"421'9721","reported_seq":9874,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.637174+0000","last_change":"2024-03-24T11:39:01.819881+0000","last_active":"2024-03-24T11:39:03.637174+0000","last_peered":"2024-03-24T11:39:03.637174+0000","last_clean":"2024-03-24T11:39:03.637174+0000","last_became_active":"2024-03-24T11:36:51.540802+0000","last_became_peered":"2024-03-24T11:36:51.540802+0000","last_unstale":"2024-03-24T11:39:03.637174+0000","last_undegraded":"2024-03-24T11:39:03.637174+0000","last_fullsized":"2024-03-24T11:39:03.637174+0000","mapping_epoch":382,"log_start":"407'5100","ondisk_log_start":"407'5100","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9589","last_scrub_stamp":"2024-03-24T11:39:01.819797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.819797+0000","objects_scrubbed":5600,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:27.376760+0000","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356974592,"num_objects":5447,"num_object_clones":0,"num_object_copies":16341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9721,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":894,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.518630+0000","last_change":"2024-03-24T11:37:34.852381+0000","last_active":"2024-03-24T11:39:03.518630+0000","last_peered":"2024-03-24T11:39:03.518630+0000","last_clean":"2024-03-24T11:39:03.518630+0000","last_became_active":"2024-03-24T11:18:56.141884+0000","last_became_peered":"2024-03-24T11:18:56.141884+0000","last_unstale":"2024-03-24T11:39:03.518630+0000","last_undegraded":"2024-03-24T11:39:03.518630+0000","last_fullsized":"2024-03-24T11:39:03.518630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:37:34.852224+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:37:34.852224+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":405,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370183+0000","last_change":"2024-03-24T11:38:46.876895+0000","last_active":"2024-03-24T11:38:54.370183+0000","last_peered":"2024-03-24T11:38:54.370183+0000","last_clean":"2024-03-24T11:38:54.370183+0000","last_became_active":"2024-03-24T11:30:35.358021+0000","last_became_peered":"2024-03-24T11:30:35.358021+0000","last_unstale":"2024-03-24T11:38:54.370183+0000","last_undegraded":"2024-03-24T11:38:54.370183+0000","last_fullsized":"2024-03-24T11:38:54.370183+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:46.876805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:46.876805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:04.648439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":811,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.672201+0000","last_change":"2024-03-24T11:37:52.553515+0000","last_active":"2024-03-24T11:39:03.672201+0000","last_peered":"2024-03-24T11:39:03.672201+0000","last_clean":"2024-03-24T11:39:03.672201+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:39:03.672201+0000","last_undegraded":"2024-03-24T11:39:03.672201+0000","last_fullsized":"2024-03-24T11:39:03.672201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:52.553402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.553402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:03.813860+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":110,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.356968+0000","last_change":"2024-03-24T11:38:12.123596+0000","last_active":"2024-03-24T11:39:04.356968+0000","last_peered":"2024-03-24T11:39:04.356968+0000","last_clean":"2024-03-24T11:39:04.356968+0000","last_became_active":"2024-03-24T11:36:42.992990+0000","last_became_peered":"2024-03-24T11:36:42.992990+0000","last_unstale":"2024-03-24T11:39:04.356968+0000","last_undegraded":"2024-03-24T11:39:04.356968+0000","last_fullsized":"2024-03-24T11:39:04.356968+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.123323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:40.144012+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"421'9745","reported_seq":9897,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.617596+0000","last_change":"2024-03-24T11:39:01.832083+0000","last_active":"2024-03-24T11:39:03.617596+0000","last_peered":"2024-03-24T11:39:03.617596+0000","last_clean":"2024-03-24T11:39:03.617596+0000","last_became_active":"2024-03-24T11:36:51.539372+0000","last_became_peered":"2024-03-24T11:36:51.539372+0000","last_unstale":"2024-03-24T11:39:03.617596+0000","last_undegraded":"2024-03-24T11:39:03.617596+0000","last_fullsized":"2024-03-24T11:39:03.617596+0000","mapping_epoch":382,"log_start":"408'5400","ondisk_log_start":"408'5400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9621","last_scrub_stamp":"2024-03-24T11:39:01.831363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.831363+0000","objects_scrubbed":5575,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.804535+0000","scrub_duration":2492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351731712,"num_objects":5367,"num_object_clones":0,"num_object_copies":16101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":483584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":105,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.360192+0000","last_change":"2024-03-24T11:38:11.115502+0000","last_active":"2024-03-24T11:38:54.360192+0000","last_peered":"2024-03-24T11:38:54.360192+0000","last_clean":"2024-03-24T11:38:54.360192+0000","last_became_active":"2024-03-24T11:36:42.470376+0000","last_became_peered":"2024-03-24T11:36:42.470376+0000","last_unstale":"2024-03-24T11:38:54.360192+0000","last_undegraded":"2024-03-24T11:38:54.360192+0000","last_fullsized":"2024-03-24T11:38:54.360192+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:11.115390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.115390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:37.708874+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"421'24","reported_seq":482,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:02.038610+0000","last_change":"2024-03-24T11:38:52.571057+0000","last_active":"2024-03-24T11:39:02.038610+0000","last_peered":"2024-03-24T11:39:02.038610+0000","last_clean":"2024-03-24T11:39:02.038610+0000","last_became_active":"2024-03-24T11:36:58.593181+0000","last_became_peered":"2024-03-24T11:36:58.593181+0000","last_unstale":"2024-03-24T11:39:02.038610+0000","last_undegraded":"2024-03-24T11:39:02.038610+0000","last_fullsized":"2024-03-24T11:39:02.038610+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:38:52.570919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.570919+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.066756+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":88,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.376462+0000","last_change":"2024-03-24T11:37:47.674201+0000","last_active":"2024-03-24T11:38:54.376462+0000","last_peered":"2024-03-24T11:38:54.376462+0000","last_clean":"2024-03-24T11:38:54.376462+0000","last_became_active":"2024-03-24T11:36:58.598204+0000","last_became_peered":"2024-03-24T11:36:58.598204+0000","last_unstale":"2024-03-24T11:38:54.376462+0000","last_undegraded":"2024-03-24T11:38:54.376462+0000","last_fullsized":"2024-03-24T11:38:54.376462+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:47.673879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:37:47.673879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:05.076109+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":760,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.361575+0000","last_change":"2024-03-24T11:33:42.747143+0000","last_active":"2024-03-24T11:39:04.361575+0000","last_peered":"2024-03-24T11:39:04.361575+0000","last_clean":"2024-03-24T11:39:04.361575+0000","last_became_active":"2024-03-24T11:18:59.706645+0000","last_became_peered":"2024-03-24T11:18:59.706645+0000","last_unstale":"2024-03-24T11:39:04.361575+0000","last_undegraded":"2024-03-24T11:39:04.361575+0000","last_fullsized":"2024-03-24T11:39:04.361575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.746752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.746752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"421'9582","reported_seq":9743,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.625306+0000","last_change":"2024-03-24T11:38:57.445535+0000","last_active":"2024-03-24T11:39:03.625306+0000","last_peered":"2024-03-24T11:39:03.625306+0000","last_clean":"2024-03-24T11:39:03.625306+0000","last_became_active":"2024-03-24T11:36:58.587355+0000","last_became_peered":"2024-03-24T11:36:58.587355+0000","last_unstale":"2024-03-24T11:39:03.625306+0000","last_undegraded":"2024-03-24T11:39:03.625306+0000","last_fullsized":"2024-03-24T11:39:03.625306+0000","mapping_epoch":387,"log_start":"406'4900","ondisk_log_start":"406'4900","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9159","last_scrub_stamp":"2024-03-24T11:38:57.445359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.445359+0000","objects_scrubbed":5752,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:26.730608+0000","scrub_duration":1611,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346161152,"num_objects":5282,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9582,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"388'11","reported_seq":551,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370358+0000","last_change":"2024-03-24T11:37:58.943704+0000","last_active":"2024-03-24T11:38:54.370358+0000","last_peered":"2024-03-24T11:38:54.370358+0000","last_clean":"2024-03-24T11:38:54.370358+0000","last_became_active":"2024-03-24T11:36:58.588094+0000","last_became_peered":"2024-03-24T11:36:58.588094+0000","last_unstale":"2024-03-24T11:38:54.370358+0000","last_undegraded":"2024-03-24T11:38:54.370358+0000","last_fullsized":"2024-03-24T11:38:54.370358+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'11","last_scrub_stamp":"2024-03-24T11:37:58.943622+0000","last_deep_scrub":"357'10","last_deep_scrub_stamp":"2024-03-24T11:35:53.386928+0000","last_clean_scrub_stamp":"2024-03-24T11:37:58.943622+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:13.122646+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"421'46","reported_seq":866,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.022417+0000","last_change":"2024-03-24T11:38:00.140302+0000","last_active":"2024-03-24T11:39:03.022417+0000","last_peered":"2024-03-24T11:39:03.022417+0000","last_clean":"2024-03-24T11:39:03.022417+0000","last_became_active":"2024-03-24T11:18:56.142258+0000","last_became_peered":"2024-03-24T11:18:56.142258+0000","last_unstale":"2024-03-24T11:39:03.022417+0000","last_undegraded":"2024-03-24T11:39:03.022417+0000","last_fullsized":"2024-03-24T11:39:03.022417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:38:00.140160+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:38:00.140160+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:10.111079+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":294,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.373407+0000","last_change":"2024-03-24T11:37:37.707970+0000","last_active":"2024-03-24T11:38:54.373407+0000","last_peered":"2024-03-24T11:38:54.373407+0000","last_clean":"2024-03-24T11:38:54.373407+0000","last_became_active":"2024-03-24T11:36:58.595352+0000","last_became_peered":"2024-03-24T11:36:58.595352+0000","last_unstale":"2024-03-24T11:38:54.373407+0000","last_undegraded":"2024-03-24T11:38:54.373407+0000","last_fullsized":"2024-03-24T11:38:54.373407+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:37:37.707835+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.707835+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":790,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368898+0000","last_change":"2024-03-24T11:38:04.034506+0000","last_active":"2024-03-24T11:38:54.368898+0000","last_peered":"2024-03-24T11:38:54.368898+0000","last_clean":"2024-03-24T11:38:54.368898+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:38:54.368898+0000","last_undegraded":"2024-03-24T11:38:54.368898+0000","last_fullsized":"2024-03-24T11:38:54.368898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:04.034411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.034411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:30.306124+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":798,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.364003+0000","last_change":"2024-03-24T11:35:53.799172+0000","last_active":"2024-03-24T11:39:04.364003+0000","last_peered":"2024-03-24T11:39:04.364003+0000","last_clean":"2024-03-24T11:39:04.364003+0000","last_became_active":"2024-03-24T11:18:56.142112+0000","last_became_peered":"2024-03-24T11:18:56.142112+0000","last_unstale":"2024-03-24T11:39:04.364003+0000","last_undegraded":"2024-03-24T11:39:04.364003+0000","last_fullsized":"2024-03-24T11:39:04.364003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:35:53.797708+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:35:53.797708+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":817,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368849+0000","last_change":"2024-03-24T11:38:00.904773+0000","last_active":"2024-03-24T11:38:54.368849+0000","last_peered":"2024-03-24T11:38:54.368849+0000","last_clean":"2024-03-24T11:38:54.368849+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:38:54.368849+0000","last_undegraded":"2024-03-24T11:38:54.368849+0000","last_fullsized":"2024-03-24T11:38:54.368849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:00.904671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:00.904671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:11.738243+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":785,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369928+0000","last_change":"2024-03-24T11:37:37.114801+0000","last_active":"2024-03-24T11:38:54.369928+0000","last_peered":"2024-03-24T11:38:54.369928+0000","last_clean":"2024-03-24T11:38:54.369928+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:38:54.369928+0000","last_undegraded":"2024-03-24T11:38:54.369928+0000","last_fullsized":"2024-03-24T11:38:54.369928+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:37.114698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.114698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:57.697360+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":419,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369356+0000","last_change":"2024-03-24T11:37:54.040251+0000","last_active":"2024-03-24T11:38:54.369356+0000","last_peered":"2024-03-24T11:38:54.369356+0000","last_clean":"2024-03-24T11:38:54.369356+0000","last_became_active":"2024-03-24T11:36:58.574704+0000","last_became_peered":"2024-03-24T11:36:58.574704+0000","last_unstale":"2024-03-24T11:38:54.369356+0000","last_undegraded":"2024-03-24T11:38:54.369356+0000","last_fullsized":"2024-03-24T11:38:54.369356+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:54.040131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.040131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:08.983328+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"421'25","reported_seq":835,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:02.147205+0000","last_change":"2024-03-24T11:38:57.281113+0000","last_active":"2024-03-24T11:39:02.147205+0000","last_peered":"2024-03-24T11:39:02.147205+0000","last_clean":"2024-03-24T11:39:02.147205+0000","last_became_active":"2024-03-24T11:18:56.142674+0000","last_became_peered":"2024-03-24T11:18:56.142674+0000","last_unstale":"2024-03-24T11:39:02.147205+0000","last_undegraded":"2024-03-24T11:39:02.147205+0000","last_fullsized":"2024-03-24T11:39:02.147205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:38:57.280974+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.280974+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:25.866071+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":88,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369804+0000","last_change":"2024-03-24T11:37:42.909195+0000","last_active":"2024-03-24T11:38:54.369804+0000","last_peered":"2024-03-24T11:38:54.369804+0000","last_clean":"2024-03-24T11:38:54.369804+0000","last_became_active":"2024-03-24T11:36:58.576859+0000","last_became_peered":"2024-03-24T11:36:58.576859+0000","last_unstale":"2024-03-24T11:38:54.369804+0000","last_undegraded":"2024-03-24T11:38:54.369804+0000","last_fullsized":"2024-03-24T11:38:54.369804+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:42.909043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.909043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:09.642694+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":764,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.363925+0000","last_change":"2024-03-24T11:33:32.456675+0000","last_active":"2024-03-24T11:39:04.363925+0000","last_peered":"2024-03-24T11:39:04.363925+0000","last_clean":"2024-03-24T11:39:04.363925+0000","last_became_active":"2024-03-24T11:18:57.693857+0000","last_became_peered":"2024-03-24T11:18:57.693857+0000","last_unstale":"2024-03-24T11:39:04.363925+0000","last_undegraded":"2024-03-24T11:39:04.363925+0000","last_fullsized":"2024-03-24T11:39:04.363925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:32.456539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:33:32.456539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":813,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.680234+0000","last_change":"2024-03-24T11:39:02.356496+0000","last_active":"2024-03-24T11:39:03.680234+0000","last_peered":"2024-03-24T11:39:03.680234+0000","last_clean":"2024-03-24T11:39:03.680234+0000","last_became_active":"2024-03-24T11:36:58.594758+0000","last_became_peered":"2024-03-24T11:36:58.594758+0000","last_unstale":"2024-03-24T11:39:03.680234+0000","last_undegraded":"2024-03-24T11:39:03.680234+0000","last_fullsized":"2024-03-24T11:39:03.680234+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:02.355825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.355825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:22.376609+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":847,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.287981+0000","last_change":"2024-03-24T11:38:11.883483+0000","last_active":"2024-03-24T11:39:03.287981+0000","last_peered":"2024-03-24T11:39:03.287981+0000","last_clean":"2024-03-24T11:39:03.287981+0000","last_became_active":"2024-03-24T11:37:33.878054+0000","last_became_peered":"2024-03-24T11:37:33.878054+0000","last_unstale":"2024-03-24T11:39:03.287981+0000","last_undegraded":"2024-03-24T11:39:03.287981+0000","last_fullsized":"2024-03-24T11:39:03.287981+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:11.883396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.883396+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:19.987915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":793,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.360068+0000","last_change":"2024-03-24T11:36:55.793667+0000","last_active":"2024-03-24T11:38:54.360068+0000","last_peered":"2024-03-24T11:38:54.360068+0000","last_clean":"2024-03-24T11:38:54.360068+0000","last_became_active":"2024-03-24T11:18:53.682533+0000","last_became_peered":"2024-03-24T11:18:53.682533+0000","last_unstale":"2024-03-24T11:38:54.360068+0000","last_undegraded":"2024-03-24T11:38:54.360068+0000","last_fullsized":"2024-03-24T11:38:54.360068+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:55.793540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:55.793540+0000","last_clean_scrub_stamp":"2024-03-24T11:36:55.793540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":1023,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.364493+0000","last_change":"2024-03-24T11:36:54.050391+0000","last_active":"2024-03-24T11:39:04.364493+0000","last_peered":"2024-03-24T11:39:04.364493+0000","last_clean":"2024-03-24T11:39:04.364493+0000","last_became_active":"2024-03-24T11:18:57.700894+0000","last_became_peered":"2024-03-24T11:18:57.700894+0000","last_unstale":"2024-03-24T11:39:04.364493+0000","last_undegraded":"2024-03-24T11:39:04.364493+0000","last_fullsized":"2024-03-24T11:39:04.364493+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:54.050291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:54.050291+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":96,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.673934+0000","last_change":"2024-03-24T11:38:50.540685+0000","last_active":"2024-03-24T11:39:03.673934+0000","last_peered":"2024-03-24T11:39:03.673934+0000","last_clean":"2024-03-24T11:39:03.673934+0000","last_became_active":"2024-03-24T11:36:58.580967+0000","last_became_peered":"2024-03-24T11:36:58.580967+0000","last_unstale":"2024-03-24T11:39:03.673934+0000","last_undegraded":"2024-03-24T11:39:03.673934+0000","last_fullsized":"2024-03-24T11:39:03.673934+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:50.540148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:38:50.540148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:10.416657+0000","scrub_duration":289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":813,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.360288+0000","last_change":"2024-03-24T11:33:19.230202+0000","last_active":"2024-03-24T11:39:04.360288+0000","last_peered":"2024-03-24T11:39:04.360288+0000","last_clean":"2024-03-24T11:39:04.360288+0000","last_became_active":"2024-03-24T11:18:56.140979+0000","last_became_peered":"2024-03-24T11:18:56.140979+0000","last_unstale":"2024-03-24T11:39:04.360288+0000","last_undegraded":"2024-03-24T11:39:04.360288+0000","last_fullsized":"2024-03-24T11:39:04.360288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:19.229920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.229920+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":90,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.672116+0000","last_change":"2024-03-24T11:38:59.220381+0000","last_active":"2024-03-24T11:39:03.672116+0000","last_peered":"2024-03-24T11:39:03.672116+0000","last_clean":"2024-03-24T11:39:03.672116+0000","last_became_active":"2024-03-24T11:36:58.580909+0000","last_became_peered":"2024-03-24T11:36:58.580909+0000","last_unstale":"2024-03-24T11:39:03.672116+0000","last_undegraded":"2024-03-24T11:39:03.672116+0000","last_fullsized":"2024-03-24T11:39:03.672116+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.220270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.220270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:19.062377+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":1043,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370135+0000","last_change":"2024-03-24T11:38:08.961040+0000","last_active":"2024-03-24T11:38:54.370135+0000","last_peered":"2024-03-24T11:38:54.370135+0000","last_clean":"2024-03-24T11:38:54.370135+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:38:54.370135+0000","last_undegraded":"2024-03-24T11:38:54.370135+0000","last_fullsized":"2024-03-24T11:38:54.370135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:38:08.960909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.960909+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.998912+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":773,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370861+0000","last_change":"2024-03-24T11:36:48.755290+0000","last_active":"2024-03-24T11:38:54.370861+0000","last_peered":"2024-03-24T11:38:54.370861+0000","last_clean":"2024-03-24T11:38:54.370861+0000","last_became_active":"2024-03-24T11:18:59.717216+0000","last_became_peered":"2024-03-24T11:18:59.717216+0000","last_unstale":"2024-03-24T11:38:54.370861+0000","last_undegraded":"2024-03-24T11:38:54.370861+0000","last_fullsized":"2024-03-24T11:38:54.370861+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:48.755147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:48.755147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'32","reported_seq":489,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.228094+0000","last_change":"2024-03-24T11:37:32.834586+0000","last_active":"2024-03-24T11:39:03.228094+0000","last_peered":"2024-03-24T11:39:03.228094+0000","last_clean":"2024-03-24T11:39:03.228094+0000","last_became_active":"2024-03-24T11:36:58.586291+0000","last_became_peered":"2024-03-24T11:36:58.586291+0000","last_unstale":"2024-03-24T11:39:03.228094+0000","last_undegraded":"2024-03-24T11:39:03.228094+0000","last_fullsized":"2024-03-24T11:39:03.228094+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:37:32.834444+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:37:32.834444+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":86,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:59.070178+0000","last_change":"2024-03-24T11:38:59.070201+0000","last_active":"2024-03-24T11:38:59.070178+0000","last_peered":"2024-03-24T11:38:59.070178+0000","last_clean":"2024-03-24T11:38:59.070178+0000","last_became_active":"2024-03-24T11:36:58.574428+0000","last_became_peered":"2024-03-24T11:36:58.574428+0000","last_unstale":"2024-03-24T11:38:59.070178+0000","last_undegraded":"2024-03-24T11:38:59.070178+0000","last_fullsized":"2024-03-24T11:38:59.070178+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.070123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.070123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:21.447867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":801,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.366931+0000","last_change":"2024-03-24T11:37:34.760552+0000","last_active":"2024-03-24T11:38:54.366931+0000","last_peered":"2024-03-24T11:38:54.366931+0000","last_clean":"2024-03-24T11:38:54.366931+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:38:54.366931+0000","last_undegraded":"2024-03-24T11:38:54.366931+0000","last_fullsized":"2024-03-24T11:38:54.366931+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:34.760436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:37:34.760436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":768,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.357948+0000","last_change":"2024-03-24T11:37:40.087780+0000","last_active":"2024-03-24T11:39:04.357948+0000","last_peered":"2024-03-24T11:39:04.357948+0000","last_clean":"2024-03-24T11:39:04.357948+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:39:04.357948+0000","last_undegraded":"2024-03-24T11:39:04.357948+0000","last_fullsized":"2024-03-24T11:39:04.357948+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.086223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.086223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":831,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:02.675078+0000","last_change":"2024-03-24T11:37:55.694778+0000","last_active":"2024-03-24T11:39:02.675078+0000","last_peered":"2024-03-24T11:39:02.675078+0000","last_clean":"2024-03-24T11:39:02.675078+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:39:02.675078+0000","last_undegraded":"2024-03-24T11:39:02.675078+0000","last_fullsized":"2024-03-24T11:39:02.675078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:37:55.694628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.694628+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:03.393716+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":406,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.364183+0000","last_change":"2024-03-24T11:37:51.172948+0000","last_active":"2024-03-24T11:39:04.364183+0000","last_peered":"2024-03-24T11:39:04.364183+0000","last_clean":"2024-03-24T11:39:04.364183+0000","last_became_active":"2024-03-24T11:36:58.575459+0000","last_became_peered":"2024-03-24T11:36:58.575459+0000","last_unstale":"2024-03-24T11:39:04.364183+0000","last_undegraded":"2024-03-24T11:39:04.364183+0000","last_fullsized":"2024-03-24T11:39:04.364183+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.172813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.172813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:18.315597+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":764,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.361380+0000","last_change":"2024-03-24T11:33:26.388166+0000","last_active":"2024-03-24T11:39:04.361380+0000","last_peered":"2024-03-24T11:39:04.361380+0000","last_clean":"2024-03-24T11:39:04.361380+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:39:04.361380+0000","last_undegraded":"2024-03-24T11:39:04.361380+0000","last_fullsized":"2024-03-24T11:39:04.361380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:26.387703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:33:26.387703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":781,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.359649+0000","last_change":"2024-03-24T11:38:50.433428+0000","last_active":"2024-03-24T11:38:54.359649+0000","last_peered":"2024-03-24T11:38:54.359649+0000","last_clean":"2024-03-24T11:38:54.359649+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:38:54.359649+0000","last_undegraded":"2024-03-24T11:38:54.359649+0000","last_fullsized":"2024-03-24T11:38:54.359649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:50.432934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:50.432934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.785937+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":811,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.359849+0000","last_change":"2024-03-24T11:33:40.546128+0000","last_active":"2024-03-24T11:39:04.359849+0000","last_peered":"2024-03-24T11:39:04.359849+0000","last_clean":"2024-03-24T11:39:04.359849+0000","last_became_active":"2024-03-24T11:18:56.141516+0000","last_became_peered":"2024-03-24T11:18:56.141516+0000","last_unstale":"2024-03-24T11:39:04.359849+0000","last_undegraded":"2024-03-24T11:39:04.359849+0000","last_fullsized":"2024-03-24T11:39:04.359849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:33:40.545997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:40.545997+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":801,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370157+0000","last_change":"2024-03-24T11:38:47.913620+0000","last_active":"2024-03-24T11:38:54.370157+0000","last_peered":"2024-03-24T11:38:54.370157+0000","last_clean":"2024-03-24T11:38:54.370157+0000","last_became_active":"2024-03-24T11:18:53.684535+0000","last_became_peered":"2024-03-24T11:18:53.684535+0000","last_unstale":"2024-03-24T11:38:54.370157+0000","last_undegraded":"2024-03-24T11:38:54.370157+0000","last_fullsized":"2024-03-24T11:38:54.370157+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.913497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.913497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:49.332361+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":88,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368749+0000","last_change":"2024-03-24T11:37:01.743777+0000","last_active":"2024-03-24T11:38:54.368749+0000","last_peered":"2024-03-24T11:38:54.368749+0000","last_clean":"2024-03-24T11:38:54.368749+0000","last_became_active":"2024-03-24T11:36:58.577143+0000","last_became_peered":"2024-03-24T11:36:58.577143+0000","last_unstale":"2024-03-24T11:38:54.368749+0000","last_undegraded":"2024-03-24T11:38:54.368749+0000","last_fullsized":"2024-03-24T11:38:54.368749+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.743684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.743684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":768,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.361649+0000","last_change":"2024-03-24T11:36:48.160280+0000","last_active":"2024-03-24T11:39:04.361649+0000","last_peered":"2024-03-24T11:39:04.361649+0000","last_clean":"2024-03-24T11:39:04.361649+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:39:04.361649+0000","last_undegraded":"2024-03-24T11:39:04.361649+0000","last_fullsized":"2024-03-24T11:39:04.361649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:48.160164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:48.160164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"421'14","reported_seq":796,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.357236+0000","last_change":"2024-03-24T11:38:54.224427+0000","last_active":"2024-03-24T11:39:04.357236+0000","last_peered":"2024-03-24T11:39:04.357236+0000","last_clean":"2024-03-24T11:39:04.357236+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:39:04.357236+0000","last_undegraded":"2024-03-24T11:39:04.357236+0000","last_fullsized":"2024-03-24T11:39:04.357236+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:38:54.222100+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:38:54.222100+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:06.658198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":801,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.373695+0000","last_change":"2024-03-24T11:38:53.264039+0000","last_active":"2024-03-24T11:38:54.373695+0000","last_peered":"2024-03-24T11:38:54.373695+0000","last_clean":"2024-03-24T11:38:54.373695+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:38:54.373695+0000","last_undegraded":"2024-03-24T11:38:54.373695+0000","last_fullsized":"2024-03-24T11:38:54.373695+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.263761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.263761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:18.212346+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":815,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.370412+0000","last_change":"2024-03-24T11:38:53.997782+0000","last_active":"2024-03-24T11:38:54.370412+0000","last_peered":"2024-03-24T11:38:54.370412+0000","last_clean":"2024-03-24T11:38:54.370412+0000","last_became_active":"2024-03-24T11:36:58.577278+0000","last_became_peered":"2024-03-24T11:36:58.577278+0000","last_unstale":"2024-03-24T11:38:54.370412+0000","last_undegraded":"2024-03-24T11:38:54.370412+0000","last_fullsized":"2024-03-24T11:38:54.370412+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.997698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.997698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:58.371734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":89,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.359897+0000","last_change":"2024-03-24T11:38:06.148171+0000","last_active":"2024-03-24T11:39:04.359897+0000","last_peered":"2024-03-24T11:39:04.359897+0000","last_clean":"2024-03-24T11:39:04.359897+0000","last_became_active":"2024-03-24T11:36:58.574827+0000","last_became_peered":"2024-03-24T11:36:58.574827+0000","last_unstale":"2024-03-24T11:39:04.359897+0000","last_undegraded":"2024-03-24T11:39:04.359897+0000","last_fullsized":"2024-03-24T11:39:04.359897+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:06.147813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:06.147813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:35.811713+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"421'32","reported_seq":862,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.459248+0000","last_change":"2024-03-24T11:38:11.032460+0000","last_active":"2024-03-24T11:39:03.459248+0000","last_peered":"2024-03-24T11:39:03.459248+0000","last_clean":"2024-03-24T11:39:03.459248+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:39:03.459248+0000","last_undegraded":"2024-03-24T11:39:03.459248+0000","last_fullsized":"2024-03-24T11:39:03.459248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:38:11.032318+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.032318+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:24.858658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":780,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.357050+0000","last_change":"2024-03-24T11:38:01.211155+0000","last_active":"2024-03-24T11:39:04.357050+0000","last_peered":"2024-03-24T11:39:04.357050+0000","last_clean":"2024-03-24T11:39:04.357050+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:39:04.357050+0000","last_undegraded":"2024-03-24T11:39:04.357050+0000","last_fullsized":"2024-03-24T11:39:04.357050+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.211052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.211052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:14.127032+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":787,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.371428+0000","last_change":"2024-03-24T11:37:59.968770+0000","last_active":"2024-03-24T11:38:54.371428+0000","last_peered":"2024-03-24T11:38:54.371428+0000","last_clean":"2024-03-24T11:38:54.371428+0000","last_became_active":"2024-03-24T11:18:57.697912+0000","last_became_peered":"2024-03-24T11:18:57.697912+0000","last_unstale":"2024-03-24T11:38:54.371428+0000","last_undegraded":"2024-03-24T11:38:54.371428+0000","last_fullsized":"2024-03-24T11:38:54.371428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:59.968275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:37:59.968275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:26.500547+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":803,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.364977+0000","last_change":"2024-03-24T11:38:12.027553+0000","last_active":"2024-03-24T11:38:54.364977+0000","last_peered":"2024-03-24T11:38:54.364977+0000","last_clean":"2024-03-24T11:38:54.364977+0000","last_became_active":"2024-03-24T11:36:58.592355+0000","last_became_peered":"2024-03-24T11:36:58.592355+0000","last_unstale":"2024-03-24T11:38:54.364977+0000","last_undegraded":"2024-03-24T11:38:54.364977+0000","last_fullsized":"2024-03-24T11:38:54.364977+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.027430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:25.034763+0000","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'30","reported_seq":862,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.423682+0000","last_change":"2024-03-24T11:37:36.696164+0000","last_active":"2024-03-24T11:39:03.423682+0000","last_peered":"2024-03-24T11:39:03.423682+0000","last_clean":"2024-03-24T11:39:03.423682+0000","last_became_active":"2024-03-24T11:33:33.699957+0000","last_became_peered":"2024-03-24T11:33:33.699957+0000","last_unstale":"2024-03-24T11:39:03.423682+0000","last_undegraded":"2024-03-24T11:39:03.423682+0000","last_fullsized":"2024-03-24T11:39:03.423682+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:37:36.695829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:36.695829+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":820,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.366332+0000","last_change":"2024-03-24T11:37:52.677447+0000","last_active":"2024-03-24T11:38:54.366332+0000","last_peered":"2024-03-24T11:38:54.366332+0000","last_clean":"2024-03-24T11:38:54.366332+0000","last_became_active":"2024-03-24T11:36:58.593112+0000","last_became_peered":"2024-03-24T11:36:58.593112+0000","last_unstale":"2024-03-24T11:38:54.366332+0000","last_undegraded":"2024-03-24T11:38:54.366332+0000","last_fullsized":"2024-03-24T11:38:54.366332+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:37:52.677294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.677294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:06.104575+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"388'10","reported_seq":1077,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369034+0000","last_change":"2024-03-24T11:38:04.870954+0000","last_active":"2024-03-24T11:38:54.369034+0000","last_peered":"2024-03-24T11:38:54.369034+0000","last_clean":"2024-03-24T11:38:54.369034+0000","last_became_active":"2024-03-24T11:36:58.580932+0000","last_became_peered":"2024-03-24T11:36:58.580932+0000","last_unstale":"2024-03-24T11:38:54.369034+0000","last_undegraded":"2024-03-24T11:38:54.369034+0000","last_fullsized":"2024-03-24T11:38:54.369034+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'10","last_scrub_stamp":"2024-03-24T11:38:04.870822+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.870822+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:15.058342+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":444,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368870+0000","last_change":"2024-03-24T11:37:42.132295+0000","last_active":"2024-03-24T11:38:54.368870+0000","last_peered":"2024-03-24T11:38:54.368870+0000","last_clean":"2024-03-24T11:38:54.368870+0000","last_became_active":"2024-03-24T11:30:35.355222+0000","last_became_peered":"2024-03-24T11:30:35.355222+0000","last_unstale":"2024-03-24T11:38:54.368870+0000","last_undegraded":"2024-03-24T11:38:54.368870+0000","last_fullsized":"2024-03-24T11:38:54.368870+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:37:42.132154+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.132154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"421'24","reported_seq":827,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.364365+0000","last_change":"2024-03-24T11:36:59.961408+0000","last_active":"2024-03-24T11:39:04.364365+0000","last_peered":"2024-03-24T11:39:04.364365+0000","last_clean":"2024-03-24T11:39:04.364365+0000","last_became_active":"2024-03-24T11:36:58.591714+0000","last_became_peered":"2024-03-24T11:36:58.591714+0000","last_unstale":"2024-03-24T11:39:04.364365+0000","last_undegraded":"2024-03-24T11:39:04.364365+0000","last_fullsized":"2024-03-24T11:39:04.364365+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:36:59.961306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:59.961306+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":86,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.366782+0000","last_change":"2024-03-24T11:37:50.620288+0000","last_active":"2024-03-24T11:38:54.366782+0000","last_peered":"2024-03-24T11:38:54.366782+0000","last_clean":"2024-03-24T11:38:54.366782+0000","last_became_active":"2024-03-24T11:36:58.597448+0000","last_became_peered":"2024-03-24T11:36:58.597448+0000","last_unstale":"2024-03-24T11:38:54.366782+0000","last_undegraded":"2024-03-24T11:38:54.366782+0000","last_fullsized":"2024-03-24T11:38:54.366782+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:50.620171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:50.620171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:10.396489+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":91,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.360090+0000","last_change":"2024-03-24T11:37:43.066187+0000","last_active":"2024-03-24T11:39:04.360090+0000","last_peered":"2024-03-24T11:39:04.360090+0000","last_clean":"2024-03-24T11:39:04.360090+0000","last_became_active":"2024-03-24T11:36:58.574806+0000","last_became_peered":"2024-03-24T11:36:58.574806+0000","last_unstale":"2024-03-24T11:39:04.360090+0000","last_undegraded":"2024-03-24T11:39:04.360090+0000","last_fullsized":"2024-03-24T11:39:04.360090+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.066063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.066063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":764,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.363615+0000","last_change":"2024-03-24T11:35:49.842222+0000","last_active":"2024-03-24T11:39:04.363615+0000","last_peered":"2024-03-24T11:39:04.363615+0000","last_clean":"2024-03-24T11:39:04.363615+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:39:04.363615+0000","last_undegraded":"2024-03-24T11:39:04.363615+0000","last_fullsized":"2024-03-24T11:39:04.363615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:49.842117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:49.842117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"421'39","reported_seq":876,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-24T11:39:03.679912+0000","last_change":"2024-03-24T11:37:56.500706+0000","last_active":"2024-03-24T11:39:03.679912+0000","last_peered":"2024-03-24T11:39:03.679912+0000","last_clean":"2024-03-24T11:39:03.679912+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:39:03.679912+0000","last_undegraded":"2024-03-24T11:39:03.679912+0000","last_fullsized":"2024-03-24T11:39:03.679912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:37:56.500311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:56.500311+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.564703+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":799,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.360852+0000","last_change":"2024-03-24T11:36:58.781277+0000","last_active":"2024-03-24T11:38:54.360852+0000","last_peered":"2024-03-24T11:38:54.360852+0000","last_clean":"2024-03-24T11:38:54.360852+0000","last_became_active":"2024-03-24T11:36:58.590975+0000","last_became_peered":"2024-03-24T11:36:58.590975+0000","last_unstale":"2024-03-24T11:38:54.360852+0000","last_undegraded":"2024-03-24T11:38:54.360852+0000","last_fullsized":"2024-03-24T11:38:54.360852+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:58.781161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:36:58.781161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":799,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.364822+0000","last_change":"2024-03-24T11:37:43.684412+0000","last_active":"2024-03-24T11:38:54.364822+0000","last_peered":"2024-03-24T11:38:54.364822+0000","last_clean":"2024-03-24T11:38:54.364822+0000","last_became_active":"2024-03-24T11:36:58.596592+0000","last_became_peered":"2024-03-24T11:36:58.596592+0000","last_unstale":"2024-03-24T11:38:54.364822+0000","last_undegraded":"2024-03-24T11:38:54.364822+0000","last_fullsized":"2024-03-24T11:38:54.364822+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.684286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.684286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:12.427075+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":87,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368529+0000","last_change":"2024-03-24T11:37:48.869860+0000","last_active":"2024-03-24T11:38:54.368529+0000","last_peered":"2024-03-24T11:38:54.368529+0000","last_clean":"2024-03-24T11:38:54.368529+0000","last_became_active":"2024-03-24T11:36:58.575356+0000","last_became_peered":"2024-03-24T11:36:58.575356+0000","last_unstale":"2024-03-24T11:38:54.368529+0000","last_undegraded":"2024-03-24T11:38:54.368529+0000","last_fullsized":"2024-03-24T11:38:54.368529+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:48.869684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:37:48.869684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"421'9","reported_seq":447,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.361223+0000","last_change":"2024-03-24T11:36:58.602264+0000","last_active":"2024-03-24T11:39:04.361223+0000","last_peered":"2024-03-24T11:39:04.361223+0000","last_clean":"2024-03-24T11:39:04.361223+0000","last_became_active":"2024-03-24T11:36:58.586221+0000","last_became_peered":"2024-03-24T11:36:58.586221+0000","last_unstale":"2024-03-24T11:39:04.361223+0000","last_undegraded":"2024-03-24T11:39:04.361223+0000","last_fullsized":"2024-03-24T11:39:04.361223+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:36:04.265719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:04.265719+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":799,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369093+0000","last_change":"2024-03-24T11:37:41.153332+0000","last_active":"2024-03-24T11:38:54.369093+0000","last_peered":"2024-03-24T11:38:54.369093+0000","last_clean":"2024-03-24T11:38:54.369093+0000","last_became_active":"2024-03-24T11:36:58.585301+0000","last_became_peered":"2024-03-24T11:36:58.585301+0000","last_unstale":"2024-03-24T11:38:54.369093+0000","last_undegraded":"2024-03-24T11:38:54.369093+0000","last_fullsized":"2024-03-24T11:38:54.369093+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.153221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.153221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:00.046302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":88,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.366622+0000","last_change":"2024-03-24T11:37:01.675006+0000","last_active":"2024-03-24T11:38:54.366622+0000","last_peered":"2024-03-24T11:38:54.366622+0000","last_clean":"2024-03-24T11:38:54.366622+0000","last_became_active":"2024-03-24T11:36:58.598728+0000","last_became_peered":"2024-03-24T11:36:58.598728+0000","last_unstale":"2024-03-24T11:38:54.366622+0000","last_undegraded":"2024-03-24T11:38:54.366622+0000","last_fullsized":"2024-03-24T11:38:54.366622+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.674916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.674916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":781,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368322+0000","last_change":"2024-03-24T11:37:46.149252+0000","last_active":"2024-03-24T11:38:54.368322+0000","last_peered":"2024-03-24T11:38:54.368322+0000","last_clean":"2024-03-24T11:38:54.368322+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:38:54.368322+0000","last_undegraded":"2024-03-24T11:38:54.368322+0000","last_fullsized":"2024-03-24T11:38:54.368322+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:46.149176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:37:46.149176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:02.169889+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"421'21","reported_seq":469,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.362163+0000","last_change":"2024-03-24T11:38:53.240408+0000","last_active":"2024-03-24T11:39:04.362163+0000","last_peered":"2024-03-24T11:39:04.362163+0000","last_clean":"2024-03-24T11:39:04.362163+0000","last_became_active":"2024-03-24T11:36:58.580893+0000","last_became_peered":"2024-03-24T11:36:58.580893+0000","last_unstale":"2024-03-24T11:39:04.362163+0000","last_undegraded":"2024-03-24T11:39:04.362163+0000","last_fullsized":"2024-03-24T11:39:04.362163+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:53.238808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.238808+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.777399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":799,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.369982+0000","last_change":"2024-03-24T11:36:58.590875+0000","last_active":"2024-03-24T11:38:54.369982+0000","last_peered":"2024-03-24T11:38:54.369982+0000","last_clean":"2024-03-24T11:38:54.369982+0000","last_became_active":"2024-03-24T11:36:58.590154+0000","last_became_peered":"2024-03-24T11:36:58.590154+0000","last_unstale":"2024-03-24T11:38:54.369982+0000","last_undegraded":"2024-03-24T11:38:54.369982+0000","last_fullsized":"2024-03-24T11:38:54.369982+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:52.302009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:36:52.302009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":799,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368753+0000","last_change":"2024-03-24T11:37:54.088530+0000","last_active":"2024-03-24T11:38:54.368753+0000","last_peered":"2024-03-24T11:38:54.368753+0000","last_clean":"2024-03-24T11:38:54.368753+0000","last_became_active":"2024-03-24T11:37:33.876071+0000","last_became_peered":"2024-03-24T11:37:33.876071+0000","last_unstale":"2024-03-24T11:38:54.368753+0000","last_undegraded":"2024-03-24T11:38:54.368753+0000","last_fullsized":"2024-03-24T11:38:54.368753+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:54.088400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.088400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:13.266896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":781,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.360099+0000","last_change":"2024-03-24T11:37:31.701772+0000","last_active":"2024-03-24T11:38:54.360099+0000","last_peered":"2024-03-24T11:38:54.360099+0000","last_clean":"2024-03-24T11:38:54.360099+0000","last_became_active":"2024-03-24T11:18:59.713040+0000","last_became_peered":"2024-03-24T11:18:59.713040+0000","last_unstale":"2024-03-24T11:38:54.360099+0000","last_undegraded":"2024-03-24T11:38:54.360099+0000","last_fullsized":"2024-03-24T11:38:54.360099+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:31.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:31.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'39","reported_seq":829,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.359957+0000","last_change":"2024-03-24T11:37:02.002116+0000","last_active":"2024-03-24T11:39:04.359957+0000","last_peered":"2024-03-24T11:39:04.359957+0000","last_clean":"2024-03-24T11:39:04.359957+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:39:04.359957+0000","last_undegraded":"2024-03-24T11:39:04.359957+0000","last_fullsized":"2024-03-24T11:39:04.359957+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:37:02.002019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:02.002019+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":774,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.363433+0000","last_change":"2024-03-24T11:35:51.761976+0000","last_active":"2024-03-24T11:39:04.363433+0000","last_peered":"2024-03-24T11:39:04.363433+0000","last_clean":"2024-03-24T11:39:04.363433+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:39:04.363433+0000","last_undegraded":"2024-03-24T11:39:04.363433+0000","last_fullsized":"2024-03-24T11:39:04.363433+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:51.761850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:51.761850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":87,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368942+0000","last_change":"2024-03-24T11:37:00.746342+0000","last_active":"2024-03-24T11:38:54.368942+0000","last_peered":"2024-03-24T11:38:54.368942+0000","last_clean":"2024-03-24T11:38:54.368942+0000","last_became_active":"2024-03-24T11:36:58.584540+0000","last_became_peered":"2024-03-24T11:36:58.584540+0000","last_unstale":"2024-03-24T11:38:54.368942+0000","last_undegraded":"2024-03-24T11:38:54.368942+0000","last_fullsized":"2024-03-24T11:38:54.368942+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:00.746264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:00.746264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":760,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-03-24T11:39:04.361452+0000","last_change":"2024-03-24T11:36:53.164614+0000","last_active":"2024-03-24T11:39:04.361452+0000","last_peered":"2024-03-24T11:39:04.361452+0000","last_clean":"2024-03-24T11:39:04.361452+0000","last_became_active":"2024-03-24T11:19:00.314155+0000","last_became_peered":"2024-03-24T11:19:00.314155+0000","last_unstale":"2024-03-24T11:39:04.361452+0000","last_undegraded":"2024-03-24T11:39:04.361452+0000","last_fullsized":"2024-03-24T11:39:04.361452+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:53.164514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:53.164514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":489,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:39:01.936548+0000","last_change":"2024-03-24T11:37:54.965402+0000","last_active":"2024-03-24T11:39:01.936548+0000","last_peered":"2024-03-24T11:39:01.936548+0000","last_clean":"2024-03-24T11:39:01.936548+0000","last_became_active":"2024-03-24T11:36:58.580796+0000","last_became_peered":"2024-03-24T11:36:58.580796+0000","last_unstale":"2024-03-24T11:39:01.936548+0000","last_undegraded":"2024-03-24T11:39:01.936548+0000","last_fullsized":"2024-03-24T11:39:01.936548+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:37:54.965309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.965309+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":794,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.361346+0000","last_change":"2024-03-24T11:38:08.039863+0000","last_active":"2024-03-24T11:38:54.361346+0000","last_peered":"2024-03-24T11:38:54.361346+0000","last_clean":"2024-03-24T11:38:54.361346+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:38:54.361346+0000","last_undegraded":"2024-03-24T11:38:54.361346+0000","last_fullsized":"2024-03-24T11:38:54.361346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:08.039401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.039401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:21.805864+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":793,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.366472+0000","last_change":"2024-03-24T11:37:40.752135+0000","last_active":"2024-03-24T11:38:54.366472+0000","last_peered":"2024-03-24T11:38:54.366472+0000","last_clean":"2024-03-24T11:38:54.366472+0000","last_became_active":"2024-03-24T11:18:57.702960+0000","last_became_peered":"2024-03-24T11:18:57.702960+0000","last_unstale":"2024-03-24T11:38:54.366472+0000","last_undegraded":"2024-03-24T11:38:54.366472+0000","last_fullsized":"2024-03-24T11:38:54.366472+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.752016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.752016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":86,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-24T11:38:54.368345+0000","last_change":"2024-03-24T11:37:51.062312+0000","last_active":"2024-03-24T11:38:54.368345+0000","last_peered":"2024-03-24T11:38:54.368345+0000","last_clean":"2024-03-24T11:38:54.368345+0000","last_became_active":"2024-03-24T11:36:58.589661+0000","last_became_peered":"2024-03-24T11:36:58.589661+0000","last_unstale":"2024-03-24T11:38:54.368345+0000","last_undegraded":"2024-03-24T11:38:54.368345+0000","last_fullsized":"2024-03-24T11:38:54.368345+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.062184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.062184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:38:57.346693+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5607850008,"num_objects":85570,"num_object_clones":0,"num_object_copies":256710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85570,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155022,"num_write_kb":7698881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16892899328,"data_stored":16892887112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70222,"ondisk_log_size":70222,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":513,"num_read_kb":508,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":5430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1540,"num_read_kb":1297,"num_write":985,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":2492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1021,"ondisk_log_size":1021,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070710,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3182900,"kb_used_data":2425580,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":90562252,"statfs":{"total":95995035648,"available":92735746048,"internally_reserved":0,"allocated":2483793920,"data_stored":2483337477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":775420363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365051,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3448764,"kb_used_data":2790004,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":90296388,"statfs":{"total":95995035648,"available":92463501312,"internally_reserved":0,"allocated":2856964096,"data_stored":2856500146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674494918},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575125,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4024596,"kb_used_data":3136844,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":89720556,"statfs":{"total":95995035648,"available":91873849344,"internally_reserved":0,"allocated":3212128256,"data_stored":3211640063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3206140,"kb_used_data":2421044,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":90539012,"statfs":{"total":95995035648,"available":92711948288,"internally_reserved":0,"allocated":2479149056,"data_stored":2478684284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411600,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2738232,"kb_used_data":2075376,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91006920,"statfs":{"total":95995035648,"available":93191086080,"internally_reserved":0,"allocated":2125185024,"data_stored":2124724633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149961,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3217236,"kb_used_data":2401672,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":90527916,"statfs":{"total":95995035648,"available":92700585984,"internally_reserved":0,"allocated":2459312128,"data_stored":2458836704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":835123655},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659392,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2168252,"kb_used_data":1392948,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":91576900,"statfs":{"total":95995035648,"available":93774745600,"internally_reserved":0,"allocated":1426378752,"data_stored":1425914755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2469199872,"data_stored":2469199872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3174633472,"data_stored":3174629400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2112491520,"data_stored":2112487448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2423717888,"data_stored":2423717888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1410531328,"data_stored":1410531328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2833453056,"data_stored":2833448984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2468872192,"data_stored":2468872192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:39:12.252 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T11:39:12.252 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:39:12.252 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 5, 6, 7] 2024-03-24T11:39:12.253 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd out 7 2024-03-24T11:39:12.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:12.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:12.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:12.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:12.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:12.538 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:12.574 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:39:12.623 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:39:12.640 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:12.664 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:39:12.693 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:12.706 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:39:12.741 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:39:12.778 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:39:12.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:12.819 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:39:12.821 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:12.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:12.858 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:39:12.861 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:39:12.861 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:39:12.862 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:39:12.862 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:39:12.862 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:39:12.862 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:39:12.862 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:39:12.862 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:39:12.885 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:12.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:12.972 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:12.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:13.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:13.147 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:13.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:13.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:39:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:39:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:39:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:39:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:12 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3146121411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:13.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:13.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:13.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:13.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3146121411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:39:13.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:39:13.211 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3146121411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:13.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:13.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:13.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:13.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:13.421 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:13.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:13.522 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:13.553 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:13.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:13.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:13.683 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:13.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:13.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:13.823 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:13.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:13.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:13.970 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:13.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:14.071 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:14.100 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:14.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:13 smithi181 ceph-mon[33998]: pgmap v933: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 959 op/s 2024-03-24T11:39:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:13 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:39:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:13 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:39:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:13 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:39:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:13 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:39:14.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[29608]: pgmap v933: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 959 op/s 2024-03-24T11:39:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:39:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:39:14.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:39:14.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:39:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[37907]: pgmap v933: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 959 op/s 2024-03-24T11:39:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:39:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:39:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:39:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:13 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:39:14.230 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:14.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:14.333 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:14.397 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:14.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:14.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:14.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:14.540 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:14.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:14.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:14.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:39:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:39:14.681 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:14.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:14.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:14.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:14.794 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:39:14.794 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:39:14.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:14.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:14.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:14.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:14.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:15.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:15.093 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:15.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:15.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:15.235 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:15.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:15.337 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:15.362 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:39:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:39:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:15.380 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:15.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:15.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:15.522 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:15.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:15.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:15 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:39:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:15 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:39:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:15 smithi181 ceph-mon[33998]: 1.6 deep-scrub starts 2024-03-24T11:39:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:15 smithi181 ceph-mon[33998]: 1.6 deep-scrub ok 2024-03-24T11:39:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:15 smithi181 ceph-mon[33998]: pgmap v934: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:39:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:15.661 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:15.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:39:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[29608]: 1.6 deep-scrub starts 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[29608]: 1.6 deep-scrub ok 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[29608]: pgmap v934: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[37907]: 1.6 deep-scrub starts 2024-03-24T11:39:15.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[37907]: 1.6 deep-scrub ok 2024-03-24T11:39:15.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:15 smithi179 ceph-mon[37907]: pgmap v934: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T11:39:15.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:15.805 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:15.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:15.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:15.949 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:15.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:16.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:16.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:16.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:16.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T11:39:16.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:16.240 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:16.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:16.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:16.382 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:16.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:16.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:39:16.484 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:16.511 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:16.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:16.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:16.636 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:16.643 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:16.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:16 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:39:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:16 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:39:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:16 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:39:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:16 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:39:16.668 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:16.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:39:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:39:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:39:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:39:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:39:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:39:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:39:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:16 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:39:16.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:16.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:16.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:16.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:16.906 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:16.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:17.008 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.033 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:17.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:17.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:17.163 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:17.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:17.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:17.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.427 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:17.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:17.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:17.557 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:17 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:39:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:17 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:39:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:17 smithi181 ceph-mon[33998]: pgmap v935: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T11:39:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:17 smithi181 ceph-mon[33998]: 11.0 scrub starts 2024-03-24T11:39:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:17 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:39:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:17 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:39:17.660 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.697 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:17.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:17.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:39:17.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:39:17.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[29608]: pgmap v935: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T11:39:17.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[29608]: 11.0 scrub starts 2024-03-24T11:39:17.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:39:17.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:39:17.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:39:17.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:39:17.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[37907]: pgmap v935: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T11:39:17.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[37907]: 11.0 scrub starts 2024-03-24T11:39:17.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:39:17.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:17 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:39:17.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:17.824 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:17.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:17.953 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:17.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:18.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:18.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:18.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:18.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:18.215 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:18.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:18.316 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:18.355 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:18.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:18.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:18.487 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:18.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:18.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:18.617 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:18.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:18.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:18.760 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:18.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:18.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:18.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:18.900 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:18.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:19.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:19.024 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:19.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:19.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:19.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.295 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:19.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:19.396 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.426 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:19.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:19.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:19.558 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: osdmap e424: 8 total, 8 up, 7 in 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: pgmap v937: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 938 op/s 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: 11.c scrub starts 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: 11.0 scrub ok 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3326207182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:39:19.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:39:19.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3326207182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:39:19.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:19 smithi181 ceph-mon[33998]: osdmap e425: 8 total, 8 up, 7 in 2024-03-24T11:39:19.653 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:39:19.653 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:39:19.653 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: osdmap e424: 8 total, 8 up, 7 in 2024-03-24T11:39:19.653 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: pgmap v937: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 938 op/s 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: 11.c scrub starts 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: 11.0 scrub ok 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3326207182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3326207182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[29608]: osdmap e425: 8 total, 8 up, 7 in 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: osdmap e424: 8 total, 8 up, 7 in 2024-03-24T11:39:19.654 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: pgmap v937: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 938 op/s 2024-03-24T11:39:19.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: 11.c scrub starts 2024-03-24T11:39:19.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: 11.0 scrub ok 2024-03-24T11:39:19.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:19.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3326207182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:39:19.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:39:19.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3326207182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:39:19.655 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:19 smithi179 ceph-mon[37907]: osdmap e425: 8 total, 8 up, 7 in 2024-03-24T11:39:19.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.698 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:19.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:19.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:19.826 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:19.928 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:19.958 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:19.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:20.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:20.102 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:20.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:20.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:20.232 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:20.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:20.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:20.376 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:20.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:20.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:20.507 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:20.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:20.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:20.637 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:20.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:20 smithi181 ceph-mon[33998]: 6.1c scrub starts 2024-03-24T11:39:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:20 smithi181 ceph-mon[33998]: 6.1c scrub ok 2024-03-24T11:39:20.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:20 smithi179 ceph-mon[29608]: 6.1c scrub starts 2024-03-24T11:39:20.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:20 smithi179 ceph-mon[29608]: 6.1c scrub ok 2024-03-24T11:39:20.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:20 smithi179 ceph-mon[37907]: 6.1c scrub starts 2024-03-24T11:39:20.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:20 smithi179 ceph-mon[37907]: 6.1c scrub ok 2024-03-24T11:39:20.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:20.766 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:20.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:20.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:20.895 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:20.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:20.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:21.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:21.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:21.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:21.181 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:21.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.315 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:21.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:21.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.447 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:21.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:21.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:21.585 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:21.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:21.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:21 smithi179 ceph-mon[29608]: pgmap v939: 158 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T11:39:21.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:21 smithi179 ceph-mon[37907]: pgmap v939: 158 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T11:39:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.727 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:21.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:21.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:21.854 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:21 smithi181 ceph-mon[33998]: pgmap v939: 158 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-24T11:39:21.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:21.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:21.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:22.083 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:22.119 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:22.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:22.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:22.257 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:22.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:22.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:22.400 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:22.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:22.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:22.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:22.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:22.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:22.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:22.674 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:22.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:22.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:22.801 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:22.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:22.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:22.931 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:22.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:23.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:23.063 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:23.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:23.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:23.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:23.196 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:23.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:23.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:23.338 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:23.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:23.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:23.480 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:23.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[29608]: osdmap e426: 8 total, 8 up, 7 in 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[29608]: pgmap v941: 158 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 911 op/s 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[37907]: osdmap e426: 8 total, 8 up, 7 in 2024-03-24T11:39:23.530 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:23 smithi179 ceph-mon[37907]: pgmap v941: 158 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 911 op/s 2024-03-24T11:39:23.583 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:23.625 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:23.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:23.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:23.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:23.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:23.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:23.886 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:23.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:23.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:23 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:39:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:23 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:39:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:23 smithi181 ceph-mon[33998]: osdmap e426: 8 total, 8 up, 7 in 2024-03-24T11:39:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:23 smithi181 ceph-mon[33998]: pgmap v941: 158 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 911 op/s 2024-03-24T11:39:23.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:24.027 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:24.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:24.049 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:39:24.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:24.165 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:24.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:24.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:24.305 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:24.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:24.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:24.446 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:24.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:24.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:24.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:24.589 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:24.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:24.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:39:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:39:24.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:24.727 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:24.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:24.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:24.882 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:24.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:24.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.021 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:25.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:25.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:25.149 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:25.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:39:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:39:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:39:25.252 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:25.291 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:25.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.424 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:25.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:25.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:25.551 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:25 smithi181 ceph-mon[33998]: osdmap e427: 8 total, 8 up, 7 in 2024-03-24T11:39:25.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:25 smithi181 ceph-mon[33998]: pgmap v943: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-03-24T11:39:25.652 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.695 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:25.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:25 smithi179 ceph-mon[29608]: osdmap e427: 8 total, 8 up, 7 in 2024-03-24T11:39:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:25 smithi179 ceph-mon[29608]: pgmap v943: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-03-24T11:39:25.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:25.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:25 smithi179 ceph-mon[37907]: osdmap e427: 8 total, 8 up, 7 in 2024-03-24T11:39:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:25 smithi179 ceph-mon[37907]: pgmap v943: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-03-24T11:39:25.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.856 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:25.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:25.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:25.982 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:25.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:26.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:26.125 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:26.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:26.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:26.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:26.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:26.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:26.415 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:26.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:26.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:26.547 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:26.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:26.648 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:26.689 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:26.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:26.790 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:26.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:26.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:26.848 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:26.921 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:26.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:27.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:27.115 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:27.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:27.216 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:27.255 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:27.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:27.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:27.391 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:27.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:27.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29604]: 2024-03-24T11:39:27.349+0000 7f36b2553700 -1 mon.a@0(leader).osd e427 definitely_dead 0 2024-03-24T11:39:27.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:27.535 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:27.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:27.542 INFO:teuthology.orchestra.run.smithi179.stderr:marked out osd.7. 2024-03-24T11:39:27.636 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:27.660 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:27.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:27.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:27.784 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:27.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:27.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[29608]: pgmap v944: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 719 B/s rd, 0 B/s wr, 1k op/s 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3179540604' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[29608]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[37907]: pgmap v944: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 719 B/s rd, 0 B/s wr, 1k op/s 2024-03-24T11:39:27.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:39:27.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:39:27.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3179540604' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-24T11:39:27.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-24T11:39:27.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:27 smithi179 ceph-mon[37907]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-24T11:39:27.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:27 smithi181 ceph-mon[33998]: pgmap v944: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 719 B/s rd, 0 B/s wr, 1k op/s 2024-03-24T11:39:27.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:27 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:39:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:27 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:39:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:27 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3179540604' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-24T11:39:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:27 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-24T11:39:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:27 smithi181 ceph-mon[33998]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-24T11:39:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:27.910 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:27.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:28.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:28.036 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:28.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.164 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:28.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:28.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:28.290 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:28.293 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:39:28.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:28.416 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:28.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.555 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:28.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:28.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:28.684 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:28.785 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:28.800 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:28 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-24T11:39:28.800 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:28 smithi179 ceph-mon[29608]: osdmap e428: 8 total, 8 up, 6 in 2024-03-24T11:39:28.800 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:28 smithi179 ceph-mon[29608]: osdmap e429: 8 total, 8 up, 6 in 2024-03-24T11:39:28.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:28 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-24T11:39:28.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:28 smithi179 ceph-mon[37907]: osdmap e428: 8 total, 8 up, 6 in 2024-03-24T11:39:28.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:28 smithi179 ceph-mon[37907]: osdmap e429: 8 total, 8 up, 6 in 2024-03-24T11:39:28.809 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:28 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-24T11:39:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:28 smithi181 ceph-mon[33998]: osdmap e428: 8 total, 8 up, 6 in 2024-03-24T11:39:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:28 smithi181 ceph-mon[33998]: osdmap e429: 8 total, 8 up, 6 in 2024-03-24T11:39:28.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:28.934 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:28.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:28.952 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:39:28.953 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:39:29.036 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:29.060 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:29.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:29.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:29.186 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:29.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:29.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:29.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:29.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:29.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:29.446 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:29.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:29.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:29.592 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:29.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: pgmap v946: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 846 B/s rd, 0 B/s wr, 904 op/s 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: 1.2 continuing backfill to osd.2 from (83'128,83'128] MIN to 83'128 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: 1.2 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 427'11207 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 427'11277 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: osdmap e430: 8 total, 8 up, 6 in 2024-03-24T11:39:29.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: pgmap v946: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 846 B/s rd, 0 B/s wr, 904 op/s 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: 1.2 continuing backfill to osd.2 from (83'128,83'128] MIN to 83'128 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: 1.2 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 427'11207 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 427'11277 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: osdmap e430: 8 total, 8 up, 6 in 2024-03-24T11:39:29.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:39:29.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:29 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:39:29.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:29.732 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:29.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:29.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:29.866 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:29.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: pgmap v946: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 846 B/s rd, 0 B/s wr, 904 op/s 2024-03-24T11:39:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: 1.2 continuing backfill to osd.2 from (83'128,83'128] MIN to 83'128 2024-03-24T11:39:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: 1.2 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: 11.4 starting backfill to osd.1 from (0'0,0'0] MAX to 427'11207 2024-03-24T11:39:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: 11.3 starting backfill to osd.5 from (0'0,0'0] MAX to 427'11277 2024-03-24T11:39:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: osdmap e430: 8 total, 8 up, 6 in 2024-03-24T11:39:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:39:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:29 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:39:29.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:30.016 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:30.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:30.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:30.150 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:30.252 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:30.281 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:30.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:30.423 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:30.493 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:30.525 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:30.576 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:30.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.718 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:30.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:30.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:30.852 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:30.894 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[29608]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 427'11230 2024-03-24T11:39:30.894 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[29608]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 427'11202 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[29608]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 427'11242 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[29608]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 427'11159 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[29608]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 427'11357 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[29608]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[29608]: osdmap e431: 8 total, 8 up, 6 in 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[37907]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 427'11230 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[37907]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 427'11202 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[37907]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 427'11242 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[37907]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 427'11159 2024-03-24T11:39:30.895 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[37907]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 427'11357 2024-03-24T11:39:30.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[37907]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:30.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:30 smithi179 ceph-mon[37907]: osdmap e431: 8 total, 8 up, 6 in 2024-03-24T11:39:30.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:30 smithi181 ceph-mon[33998]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 427'11230 2024-03-24T11:39:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:30 smithi181 ceph-mon[33998]: 11.e starting backfill to osd.3 from (0'0,0'0] MAX to 427'11202 2024-03-24T11:39:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:30 smithi181 ceph-mon[33998]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 427'11242 2024-03-24T11:39:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:30 smithi181 ceph-mon[33998]: 11.c starting backfill to osd.5 from (0'0,0'0] MAX to 427'11159 2024-03-24T11:39:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:30 smithi181 ceph-mon[33998]: 11.5 starting backfill to osd.3 from (0'0,0'0] MAX to 427'11357 2024-03-24T11:39:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:30 smithi181 ceph-mon[33998]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 83'128 2024-03-24T11:39:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:30 smithi181 ceph-mon[33998]: osdmap e431: 8 total, 8 up, 6 in 2024-03-24T11:39:30.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T11:39:30.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:30.986 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:30.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:31.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:31.123 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:31.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:31.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:31.255 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:31.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:31.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:31.405 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:31.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:31.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:31.540 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:31.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:31.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:31.671 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:31.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:31.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:31 smithi179 ceph-mon[29608]: pgmap v950: 158 pgs: 1 active+undersized+remapped, 2 active+clean+remapped, 5 remapped+peering, 9 peering, 141 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 80 KiB/s, 18 objects/s recovering 2024-03-24T11:39:31.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:31 smithi179 ceph-mon[29608]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:39:31.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:31 smithi179 ceph-mon[29608]: osdmap e432: 8 total, 8 up, 6 in 2024-03-24T11:39:31.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:31 smithi179 ceph-mon[37907]: pgmap v950: 158 pgs: 1 active+undersized+remapped, 2 active+clean+remapped, 5 remapped+peering, 9 peering, 141 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 80 KiB/s, 18 objects/s recovering 2024-03-24T11:39:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:31 smithi179 ceph-mon[37907]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:39:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:31 smithi179 ceph-mon[37907]: osdmap e432: 8 total, 8 up, 6 in 2024-03-24T11:39:31.774 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:31.808 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:31.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:31 smithi181 ceph-mon[33998]: pgmap v950: 158 pgs: 1 active+undersized+remapped, 2 active+clean+remapped, 5 remapped+peering, 9 peering, 141 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 80 KiB/s, 18 objects/s recovering 2024-03-24T11:39:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:31 smithi181 ceph-mon[33998]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:39:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:31 smithi181 ceph-mon[33998]: osdmap e432: 8 total, 8 up, 6 in 2024-03-24T11:39:31.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:31.950 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:31.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:32.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:32.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:32.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:32.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:32.220 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:32.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:32.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:32.365 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:32.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:32.466 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:32.519 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:32.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:32.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:32.652 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:32.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:32.666 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:32.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:32.774 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:39:32.804 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:32.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:32.830 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:39:32.873 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:39:32.906 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:32.933 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:39:32.954 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:32.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:32.973 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:39:33.014 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:39:33.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:33.057 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:39:33.088 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:33.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:33.096 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:39:33.102 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:39:33.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:33.221 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:33.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:33.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:33.362 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:33.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:33.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:33.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:33.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:33.642 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:33.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:33.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:33.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:33.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:33.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:33.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:33 smithi181 ceph-mon[33998]: pgmap v952: 158 pgs: 1 active+undersized+remapped, 2 active+clean+remapped, 5 remapped+peering, 9 peering, 141 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 80 KiB/s, 18 objects/s recovering 2024-03-24T11:39:33.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:33 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/614007289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:33.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:33.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:33 smithi179 ceph-mon[29608]: pgmap v952: 158 pgs: 1 active+undersized+remapped, 2 active+clean+remapped, 5 remapped+peering, 9 peering, 141 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 80 KiB/s, 18 objects/s recovering 2024-03-24T11:39:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:33 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/614007289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:33 smithi179 ceph-mon[37907]: pgmap v952: 158 pgs: 1 active+undersized+remapped, 2 active+clean+remapped, 5 remapped+peering, 9 peering, 141 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 80 KiB/s, 18 objects/s recovering 2024-03-24T11:39:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:33 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/614007289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:34.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:34.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:34.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:34.147 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:34.192 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:34.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:34.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:34.324 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:34.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:34.373 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6] out_osds: [4, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:39:34.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:39:34.373 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-24T11:39:34.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:39:34.426 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:34.474 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:34.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:34.576 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:34.623 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:34.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:34.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:39:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:39:34.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:34.756 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:34.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:34.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:34.921 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:34.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:35.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:35.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:35.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:35.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:35.191 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:35.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:35.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:35.322 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:35.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:34 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:34 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:34 smithi181 ceph-mon[33998]: osdmap e433: 8 total, 8 up, 6 in 2024-03-24T11:39:35.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:35.460 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:39:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:39:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:39:35.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:34 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:35.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:34 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:35.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:34 smithi179 ceph-mon[29608]: osdmap e433: 8 total, 8 up, 6 in 2024-03-24T11:39:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:34 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:39:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:34 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:35.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:34 smithi179 ceph-mon[37907]: osdmap e433: 8 total, 8 up, 6 in 2024-03-24T11:39:35.461 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:35.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:35.564 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:35.612 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:35.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:35.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:35.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:35.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:35.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:35.878 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:35.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:35.980 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:36.011 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:39:36.025 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:36.055 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:36.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:36.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:36.190 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:36.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:36 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:36 smithi179 ceph-mon[29608]: pgmap v954: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 active+clean+remapped, 149 active+clean; 3.5 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1000 B/s rd, 0 B/s wr, 737 op/s; 20556/171651 objects misplaced (11.975%); 50 MiB/s, 811 objects/s recovering 2024-03-24T11:39:36.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1466628064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:36 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:36 smithi179 ceph-mon[37907]: pgmap v954: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 active+clean+remapped, 149 active+clean; 3.5 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1000 B/s rd, 0 B/s wr, 737 op/s; 20556/171651 objects misplaced (11.975%); 50 MiB/s, 811 objects/s recovering 2024-03-24T11:39:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1466628064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:36.291 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:36.329 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:36.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:36 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:36 smithi181 ceph-mon[33998]: pgmap v954: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 active+clean+remapped, 149 active+clean; 3.5 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1000 B/s rd, 0 B/s wr, 737 op/s; 20556/171651 objects misplaced (11.975%); 50 MiB/s, 811 objects/s recovering 2024-03-24T11:39:36.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1466628064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:39:36.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:36.471 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:36.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:36.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:39:36.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:36.603 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:36.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:36.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:36.737 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:36.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:36.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:36.888 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:36.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:36.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:37.021 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:37.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:37.115 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-24T11:39:37.115 INFO:teuthology.orchestra.run.smithi179.stdout: "ops": [], 2024-03-24T11:39:37.115 INFO:teuthology.orchestra.run.smithi179.stdout: "complaint_time": 30, 2024-03-24T11:39:37.115 INFO:teuthology.orchestra.run.smithi179.stdout: "num_blocked_ops": 0 2024-03-24T11:39:37.115 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-24T11:39:37.115 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:37 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:39:37.116 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:37 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1466628064' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:39:37.116 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:37 smithi179 ceph-mon[29608]: osdmap e434: 8 total, 8 up, 6 in 2024-03-24T11:39:37.116 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:37 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:39:37.116 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:37 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1466628064' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:39:37.116 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:37 smithi179 ceph-mon[37907]: osdmap e434: 8 total, 8 up, 6 in 2024-03-24T11:39:37.124 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:37.155 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:37.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:37.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:37.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:37.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:37.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:37 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:39:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:37 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1466628064' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:39:37.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:37 smithi181 ceph-mon[33998]: osdmap e434: 8 total, 8 up, 6 in 2024-03-24T11:39:37.445 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:37.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:37.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:37.584 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:37.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:37.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:37.729 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:37.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:37.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:37.872 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:37.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:37.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T11:39:37.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:38.006 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:38.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:38.139 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:38 smithi179 ceph-mon[37907]: pgmap v956: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 151 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 691 op/s; 20556/167205 objects misplaced (12.294%); 43 MiB/s, 688 objects/s recovering 2024-03-24T11:39:38.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:38 smithi179 ceph-mon[29608]: pgmap v956: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 151 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 691 op/s; 20556/167205 objects misplaced (12.294%); 43 MiB/s, 688 objects/s recovering 2024-03-24T11:39:38.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:38.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:38.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:38.401 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:38 smithi181 ceph-mon[33998]: pgmap v956: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 151 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 691 op/s; 20556/167205 objects misplaced (12.294%); 43 MiB/s, 688 objects/s recovering 2024-03-24T11:39:38.404 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:38.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.543 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:38.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:38.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:38.686 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:38.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:38.817 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:38.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:38.959 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:38.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:39.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:39.105 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:39.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:39.207 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:39.238 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:39.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:39.340 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:39.388 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:39.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:39.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:39.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:39.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:39.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:39.656 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:39.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:39.759 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:39.789 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:39.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:39.891 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:39.921 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:39.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:40.023 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:40.054 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:40.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:40.187 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:40.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:40.319 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:40.375 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:40 smithi179 ceph-mon[29608]: pgmap v957: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 151 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 721 B/s rd, 0 B/s wr, 584 op/s; 20556/167205 objects misplaced (12.294%); 36 MiB/s, 582 objects/s recovering 2024-03-24T11:39:40.375 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:40 smithi179 ceph-mon[29608]: osdmap e435: 8 total, 8 up, 6 in 2024-03-24T11:39:40.376 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:40 smithi179 ceph-mon[37907]: pgmap v957: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 151 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 721 B/s rd, 0 B/s wr, 584 op/s; 20556/167205 objects misplaced (12.294%); 36 MiB/s, 582 objects/s recovering 2024-03-24T11:39:40.376 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:40 smithi179 ceph-mon[37907]: osdmap e435: 8 total, 8 up, 6 in 2024-03-24T11:39:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:40 smithi181 ceph-mon[33998]: pgmap v957: 158 pgs: 6 active+remapped+backfilling, 1 active+remapped+backfill_wait, 151 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 721 B/s rd, 0 B/s wr, 584 op/s; 20556/167205 objects misplaced (12.294%); 36 MiB/s, 582 objects/s recovering 2024-03-24T11:39:40.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:40 smithi181 ceph-mon[33998]: osdmap e435: 8 total, 8 up, 6 in 2024-03-24T11:39:40.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:40.454 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:40.462 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:40.546 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:40.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:40.587 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:40.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.721 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:40.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:40.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:40.850 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:40.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:40.978 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:40.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:41.082 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:41.125 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:41.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:41.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:41.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:41.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:41.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:41.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:41.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:41.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:39:41.530 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:41.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:41.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:41.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:41 smithi181 ceph-mon[33998]: osdmap e436: 8 total, 8 up, 6 in 2024-03-24T11:39:41.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:41 smithi181 ceph-mon[33998]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 151 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s; 14469/152706 objects misplaced (9.475%); 46 MiB/s, 735 objects/s recovering 2024-03-24T11:39:41.662 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:41.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:41.664 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:39:41.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:41 smithi179 ceph-mon[29608]: osdmap e436: 8 total, 8 up, 6 in 2024-03-24T11:39:41.664 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:41 smithi179 ceph-mon[29608]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 151 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s; 14469/152706 objects misplaced (9.475%); 46 MiB/s, 735 objects/s recovering 2024-03-24T11:39:41.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:41 smithi179 ceph-mon[37907]: osdmap e436: 8 total, 8 up, 6 in 2024-03-24T11:39:41.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:41 smithi179 ceph-mon[37907]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 151 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s; 14469/152706 objects misplaced (9.475%); 46 MiB/s, 735 objects/s recovering 2024-03-24T11:39:41.687 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:39:41.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:41.807 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:41.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:41.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:41.953 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:41.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:42.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:42.084 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:42.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:42.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:42.226 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:42.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:42.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:42.359 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:42.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:42.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:42.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:42.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:42.503 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":960,"stamp":"2024-03-24T11:39:40.473680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3323468150,"num_objects":50902,"num_object_clones":0,"num_object_copies":152706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14469,"num_objects_unfound":0,"num_objects_dirty":50902,"num_whiteouts":0,"num_read":2114,"num_read_kb":1866,"num_write":190896,"num_write_kb":7698926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8748,"num_bytes_recovered":560273076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61073,"ondisk_log_size":61073,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":13654688,"kb_used_data":8923488,"kb_used_omap":9,"kb_used_meta":4730934,"kb_avail":548816224,"statfs":{"total":575970213888,"available":561987813376,"internally_reserved":0,"allocated":9137651712,"data_stored":9134327573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":4844477089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,8,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-413859840,"num_objects":-6315,"num_object_clones":0,"num_object_copies":-18945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6087,"num_objects_unfound":0,"num_objects_dirty":-6315,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6315,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4413,"num_bytes_recovered":289210368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003559"},"pg_stats":[{"pgid":"11.3","version":"434'11838","reported_seq":18537,"reported_epoch":434,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T11:39:38.628946+0000","last_change":"2024-03-24T11:39:29.469323+0000","last_active":"2024-03-24T11:39:38.628946+0000","last_peered":"2024-03-24T11:39:38.628946+0000","last_clean":"2024-03-24T11:39:27.515433+0000","last_became_active":"2024-03-24T11:39:29.440742+0000","last_became_peered":"2024-03-24T11:39:29.440742+0000","last_unstale":"2024-03-24T11:39:38.628946+0000","last_undegraded":"2024-03-24T11:39:38.628946+0000","last_fullsized":"2024-03-24T11:39:38.628946+0000","mapping_epoch":429,"log_start":"415'7830","ondisk_log_start":"415'7830","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9353","last_scrub_stamp":"2024-03-24T11:38:58.630837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:58.630837+0000","objects_scrubbed":5739,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209190912,"num_objects":3192,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":195,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11838,"num_write_kb":480960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3258,"num_bytes_recovered":213516288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"1,6,7","objects":3192}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"421'30","reported_seq":507,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381842+0000","last_change":"2024-03-24T11:38:10.116849+0000","last_active":"2024-03-24T11:39:39.381842+0000","last_peered":"2024-03-24T11:39:39.381842+0000","last_clean":"2024-03-24T11:39:39.381842+0000","last_became_active":"2024-03-24T11:36:58.591623+0000","last_became_peered":"2024-03-24T11:36:58.591623+0000","last_unstale":"2024-03-24T11:39:39.381842+0000","last_undegraded":"2024-03-24T11:39:39.381842+0000","last_fullsized":"2024-03-24T11:39:39.381842+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:10.116699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.116699+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":803,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381284+0000","last_change":"2024-03-24T11:36:58.586904+0000","last_active":"2024-03-24T11:39:39.381284+0000","last_peered":"2024-03-24T11:39:39.381284+0000","last_clean":"2024-03-24T11:39:39.381284+0000","last_became_active":"2024-03-24T11:36:58.586602+0000","last_became_peered":"2024-03-24T11:36:58.586602+0000","last_unstale":"2024-03-24T11:39:39.381284+0000","last_undegraded":"2024-03-24T11:39:39.381284+0000","last_fullsized":"2024-03-24T11:39:39.381284+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:25.353054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:33:25.353054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"429'10","reported_seq":1112,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.012597+0000","last_change":"2024-03-24T11:39:28.587290+0000","last_active":"2024-03-24T11:39:36.012597+0000","last_peered":"2024-03-24T11:39:36.012597+0000","last_clean":"2024-03-24T11:39:36.012597+0000","last_became_active":"2024-03-24T11:39:28.568543+0000","last_became_peered":"2024-03-24T11:39:28.568543+0000","last_unstale":"2024-03-24T11:39:36.012597+0000","last_undegraded":"2024-03-24T11:39:36.012597+0000","last_fullsized":"2024-03-24T11:39:36.012597+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'9","last_scrub_stamp":"2024-03-24T11:39:08.307817+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:39:08.307817+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:18.651883+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":113,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.003100+0000","last_change":"2024-03-24T11:38:01.792439+0000","last_active":"2024-03-24T11:39:36.003100+0000","last_peered":"2024-03-24T11:39:36.003100+0000","last_clean":"2024-03-24T11:39:36.003100+0000","last_became_active":"2024-03-24T11:36:58.590765+0000","last_became_peered":"2024-03-24T11:36:58.590765+0000","last_unstale":"2024-03-24T11:39:36.003100+0000","last_undegraded":"2024-03-24T11:39:36.003100+0000","last_fullsized":"2024-03-24T11:39:36.003100+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.792132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.792132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":890,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.388961+0000","last_change":"2024-03-24T11:39:31.408536+0000","last_active":"2024-03-24T11:39:39.388961+0000","last_peered":"2024-03-24T11:39:39.388961+0000","last_clean":"2024-03-24T11:39:39.388961+0000","last_became_active":"2024-03-24T11:39:31.407869+0000","last_became_peered":"2024-03-24T11:39:31.407869+0000","last_unstale":"2024-03-24T11:39:39.388961+0000","last_undegraded":"2024-03-24T11:39:39.388961+0000","last_fullsized":"2024-03-24T11:39:39.388961+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:57.972898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:57.972898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"435'11850","reported_seq":11966,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.622800+0000","last_change":"2024-03-24T11:36:51.538337+0000","last_active":"2024-03-24T11:39:39.622800+0000","last_peered":"2024-03-24T11:39:39.622800+0000","last_clean":"2024-03-24T11:39:39.622800+0000","last_became_active":"2024-03-24T11:36:51.537990+0000","last_became_peered":"2024-03-24T11:36:51.537990+0000","last_unstale":"2024-03-24T11:39:39.622800+0000","last_undegraded":"2024-03-24T11:39:39.622800+0000","last_fullsized":"2024-03-24T11:39:39.622800+0000","mapping_epoch":382,"log_start":"417'8294","ondisk_log_start":"417'8294","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":3556,"log_dups_size":0,"ondisk_log_size":3556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206569472,"num_objects":3152,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11850,"num_write_kb":480064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"421'53","reported_seq":584,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.012353+0000","last_change":"2024-03-24T11:38:04.488873+0000","last_active":"2024-03-24T11:39:36.012353+0000","last_peered":"2024-03-24T11:39:36.012353+0000","last_clean":"2024-03-24T11:39:36.012353+0000","last_became_active":"2024-03-24T11:36:58.580651+0000","last_became_peered":"2024-03-24T11:36:58.580651+0000","last_unstale":"2024-03-24T11:39:36.012353+0000","last_undegraded":"2024-03-24T11:39:36.012353+0000","last_fullsized":"2024-03-24T11:39:36.012353+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'39","last_scrub_stamp":"2024-03-24T11:38:04.488661+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.488661+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:37.345704+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":861,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009440+0000","last_change":"2024-03-24T11:38:48.018756+0000","last_active":"2024-03-24T11:39:36.009440+0000","last_peered":"2024-03-24T11:39:36.009440+0000","last_clean":"2024-03-24T11:39:36.009440+0000","last_became_active":"2024-03-24T11:36:58.587097+0000","last_became_peered":"2024-03-24T11:36:58.587097+0000","last_unstale":"2024-03-24T11:39:36.009440+0000","last_undegraded":"2024-03-24T11:39:36.009440+0000","last_fullsized":"2024-03-24T11:39:36.009440+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:38:48.018617+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.018617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.449456+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":826,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.030464+0000","last_change":"2024-03-24T11:37:04.724933+0000","last_active":"2024-03-24T11:39:36.030464+0000","last_peered":"2024-03-24T11:39:36.030464+0000","last_clean":"2024-03-24T11:39:36.030464+0000","last_became_active":"2024-03-24T11:36:58.595572+0000","last_became_peered":"2024-03-24T11:36:58.595572+0000","last_unstale":"2024-03-24T11:39:36.030464+0000","last_undegraded":"2024-03-24T11:39:36.030464+0000","last_fullsized":"2024-03-24T11:39:36.030464+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:04.724780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:48.793614+0000","last_clean_scrub_stamp":"2024-03-24T11:37:04.724780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":139,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381350+0000","last_change":"2024-03-24T11:38:52.569860+0000","last_active":"2024-03-24T11:39:39.381350+0000","last_peered":"2024-03-24T11:39:39.381350+0000","last_clean":"2024-03-24T11:39:39.381350+0000","last_became_active":"2024-03-24T11:36:58.582225+0000","last_became_peered":"2024-03-24T11:36:58.582225+0000","last_unstale":"2024-03-24T11:39:39.381350+0000","last_undegraded":"2024-03-24T11:39:39.381350+0000","last_fullsized":"2024-03-24T11:39:39.381350+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:52.569730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.569730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.998514+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":984,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.007926+0000","last_change":"2024-03-24T11:38:03.034263+0000","last_active":"2024-03-24T11:39:36.007926+0000","last_peered":"2024-03-24T11:39:36.007926+0000","last_clean":"2024-03-24T11:39:36.007926+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:39:36.007926+0000","last_undegraded":"2024-03-24T11:39:36.007926+0000","last_fullsized":"2024-03-24T11:39:36.007926+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:38:03.034131+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:38:03.034131+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"434'11831","reported_seq":12018,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:38.613385+0000","last_change":"2024-03-24T11:38:48.786761+0000","last_active":"2024-03-24T11:39:38.613385+0000","last_peered":"2024-03-24T11:39:38.613385+0000","last_clean":"2024-03-24T11:39:38.613385+0000","last_became_active":"2024-03-24T11:36:51.529788+0000","last_became_peered":"2024-03-24T11:36:51.529788+0000","last_unstale":"2024-03-24T11:39:38.613385+0000","last_undegraded":"2024-03-24T11:39:38.613385+0000","last_fullsized":"2024-03-24T11:39:38.613385+0000","mapping_epoch":382,"log_start":"415'7822","ondisk_log_start":"415'7822","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'8678","last_scrub_stamp":"2024-03-24T11:38:48.786544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.786544+0000","objects_scrubbed":6471,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:10.325229+0000","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213712896,"num_objects":3261,"num_object_clones":0,"num_object_copies":9783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11831,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":511,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.010938+0000","last_change":"2024-03-24T11:37:31.892734+0000","last_active":"2024-03-24T11:39:36.010938+0000","last_peered":"2024-03-24T11:39:36.010938+0000","last_clean":"2024-03-24T11:39:36.010938+0000","last_became_active":"2024-03-24T11:36:58.587116+0000","last_became_peered":"2024-03-24T11:36:58.587116+0000","last_unstale":"2024-03-24T11:39:36.010938+0000","last_undegraded":"2024-03-24T11:39:36.010938+0000","last_fullsized":"2024-03-24T11:39:36.010938+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:37:31.892595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:31.892595+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":119,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.005721+0000","last_change":"2024-03-24T11:39:28.553772+0000","last_active":"2024-03-24T11:39:36.005721+0000","last_peered":"2024-03-24T11:39:36.005721+0000","last_clean":"2024-03-24T11:39:36.005721+0000","last_became_active":"2024-03-24T11:39:28.553193+0000","last_became_peered":"2024-03-24T11:39:28.553193+0000","last_unstale":"2024-03-24T11:39:36.005721+0000","last_undegraded":"2024-03-24T11:39:36.005721+0000","last_fullsized":"2024-03-24T11:39:36.005721+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.247390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.247390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:50.023392+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":818,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009739+0000","last_change":"2024-03-24T11:37:38.108397+0000","last_active":"2024-03-24T11:39:36.009739+0000","last_peered":"2024-03-24T11:39:36.009739+0000","last_clean":"2024-03-24T11:39:36.009739+0000","last_became_active":"2024-03-24T11:36:58.575297+0000","last_became_peered":"2024-03-24T11:36:58.575297+0000","last_unstale":"2024-03-24T11:39:36.009739+0000","last_undegraded":"2024-03-24T11:39:36.009739+0000","last_fullsized":"2024-03-24T11:39:36.009739+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:38.108325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:37:38.108325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":785,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381610+0000","last_change":"2024-03-24T11:33:38.472649+0000","last_active":"2024-03-24T11:39:39.381610+0000","last_peered":"2024-03-24T11:39:39.381610+0000","last_clean":"2024-03-24T11:39:39.381610+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:39:39.381610+0000","last_undegraded":"2024-03-24T11:39:39.381610+0000","last_fullsized":"2024-03-24T11:39:39.381610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:38.472418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:38.472418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"434'11791","reported_seq":14409,"reported_epoch":434,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T11:39:38.526776+0000","last_change":"2024-03-24T11:39:29.535526+0000","last_active":"2024-03-24T11:39:38.526776+0000","last_peered":"2024-03-24T11:39:38.526776+0000","last_clean":"2024-03-24T11:39:27.528618+0000","last_became_active":"2024-03-24T11:39:29.453741+0000","last_became_peered":"2024-03-24T11:39:29.453741+0000","last_unstale":"2024-03-24T11:39:38.526776+0000","last_undegraded":"2024-03-24T11:39:38.526776+0000","last_fullsized":"2024-03-24T11:39:38.526776+0000","mapping_epoch":429,"log_start":"417'8282","ondisk_log_start":"417'8282","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'10628","last_scrub_stamp":"2024-03-24T11:39:18.494156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:39:18.494156+0000","objects_scrubbed":4418,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208732160,"num_objects":3185,"num_object_clones":0,"num_object_copies":9555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2066,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11791,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1221,"num_bytes_recovered":80019456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":3185}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"421'46","reported_seq":936,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.007126+0000","last_change":"2024-03-24T11:39:28.604710+0000","last_active":"2024-03-24T11:39:36.007126+0000","last_peered":"2024-03-24T11:39:36.007126+0000","last_clean":"2024-03-24T11:39:36.007126+0000","last_became_active":"2024-03-24T11:39:28.585895+0000","last_became_peered":"2024-03-24T11:39:28.585895+0000","last_unstale":"2024-03-24T11:39:36.007126+0000","last_undegraded":"2024-03-24T11:39:36.007126+0000","last_fullsized":"2024-03-24T11:39:36.007126+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:36:56.684283+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:36:56.684283+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":811,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.383127+0000","last_change":"2024-03-24T11:39:28.589571+0000","last_active":"2024-03-24T11:39:39.383127+0000","last_peered":"2024-03-24T11:39:39.383127+0000","last_clean":"2024-03-24T11:39:39.383127+0000","last_became_active":"2024-03-24T11:39:28.587712+0000","last_became_peered":"2024-03-24T11:39:28.587712+0000","last_unstale":"2024-03-24T11:39:39.383127+0000","last_undegraded":"2024-03-24T11:39:39.383127+0000","last_fullsized":"2024-03-24T11:39:39.383127+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.005586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.005586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'9","reported_seq":182,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008760+0000","last_change":"2024-03-24T11:38:51.903763+0000","last_active":"2024-03-24T11:39:36.008760+0000","last_peered":"2024-03-24T11:39:36.008760+0000","last_clean":"2024-03-24T11:39:36.008760+0000","last_became_active":"2024-03-24T11:36:58.574227+0000","last_became_peered":"2024-03-24T11:36:58.574227+0000","last_unstale":"2024-03-24T11:39:36.008760+0000","last_undegraded":"2024-03-24T11:39:36.008760+0000","last_fullsized":"2024-03-24T11:39:36.008760+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-03-24T11:38:51.903639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:51.903639+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.344849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":808,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009106+0000","last_change":"2024-03-24T11:38:55.092421+0000","last_active":"2024-03-24T11:39:36.009106+0000","last_peered":"2024-03-24T11:39:36.009106+0000","last_clean":"2024-03-24T11:39:36.009106+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:39:36.009106+0000","last_undegraded":"2024-03-24T11:39:36.009106+0000","last_fullsized":"2024-03-24T11:39:36.009106+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:55.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:55.092309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.593383+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":975,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.007914+0000","last_change":"2024-03-24T11:37:44.069186+0000","last_active":"2024-03-24T11:39:36.007914+0000","last_peered":"2024-03-24T11:39:36.007914+0000","last_clean":"2024-03-24T11:39:36.007914+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:39:36.007914+0000","last_undegraded":"2024-03-24T11:39:36.007914+0000","last_fullsized":"2024-03-24T11:39:36.007914+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:44.069073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.069073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"434'11767","reported_seq":11944,"reported_epoch":434,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-24T11:39:38.506437+0000","last_change":"2024-03-24T11:39:29.532208+0000","last_active":"2024-03-24T11:39:38.506437+0000","last_peered":"2024-03-24T11:39:38.506437+0000","last_clean":"2024-03-24T11:39:27.537839+0000","last_became_active":"2024-03-24T11:39:29.475687+0000","last_became_peered":"2024-03-24T11:39:29.475687+0000","last_unstale":"2024-03-24T11:39:38.506437+0000","last_undegraded":"2024-03-24T11:39:38.506437+0000","last_fullsized":"2024-03-24T11:39:38.506437+0000","mapping_epoch":429,"log_start":"417'8294","ondisk_log_start":"417'8294","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'10064","last_scrub_stamp":"2024-03-24T11:39:10.342567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:39:10.342567+0000","objects_scrubbed":4935,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208601088,"num_objects":3183,"num_object_clones":0,"num_object_copies":9549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3183,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":122,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.390393+0000","last_change":"2024-03-24T11:39:28.567930+0000","last_active":"2024-03-24T11:39:39.390393+0000","last_peered":"2024-03-24T11:39:39.390393+0000","last_clean":"2024-03-24T11:39:39.390393+0000","last_became_active":"2024-03-24T11:39:28.567528+0000","last_became_peered":"2024-03-24T11:39:28.567528+0000","last_unstale":"2024-03-24T11:39:39.390393+0000","last_undegraded":"2024-03-24T11:39:39.390393+0000","last_fullsized":"2024-03-24T11:39:39.390393+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.082984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.082984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"421'42","reported_seq":886,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.383208+0000","last_change":"2024-03-24T11:39:28.610844+0000","last_active":"2024-03-24T11:39:39.383208+0000","last_peered":"2024-03-24T11:39:39.383208+0000","last_clean":"2024-03-24T11:39:39.383208+0000","last_became_active":"2024-03-24T11:39:28.588159+0000","last_became_peered":"2024-03-24T11:39:28.588159+0000","last_unstale":"2024-03-24T11:39:39.383208+0000","last_undegraded":"2024-03-24T11:39:39.383208+0000","last_fullsized":"2024-03-24T11:39:39.383208+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:37:32.916579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:32.916579+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":811,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.380947+0000","last_change":"2024-03-24T11:39:28.568155+0000","last_active":"2024-03-24T11:39:39.380947+0000","last_peered":"2024-03-24T11:39:39.380947+0000","last_clean":"2024-03-24T11:39:39.380947+0000","last_became_active":"2024-03-24T11:39:28.566304+0000","last_became_peered":"2024-03-24T11:39:28.566304+0000","last_unstale":"2024-03-24T11:39:39.380947+0000","last_undegraded":"2024-03-24T11:39:39.380947+0000","last_fullsized":"2024-03-24T11:39:39.380947+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:57.213332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.213332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.625878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":107,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.389284+0000","last_change":"2024-03-24T11:36:58.575117+0000","last_active":"2024-03-24T11:39:39.389284+0000","last_peered":"2024-03-24T11:39:39.389284+0000","last_clean":"2024-03-24T11:39:39.389284+0000","last_became_active":"2024-03-24T11:36:58.574742+0000","last_became_peered":"2024-03-24T11:36:58.574742+0000","last_unstale":"2024-03-24T11:39:39.389284+0000","last_undegraded":"2024-03-24T11:39:39.389284+0000","last_fullsized":"2024-03-24T11:39:39.389284+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:30.982770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:33:30.982770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":428,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.012692+0000","last_change":"2024-03-24T11:38:54.329326+0000","last_active":"2024-03-24T11:39:36.012692+0000","last_peered":"2024-03-24T11:39:36.012692+0000","last_clean":"2024-03-24T11:39:36.012692+0000","last_became_active":"2024-03-24T11:37:01.307977+0000","last_became_peered":"2024-03-24T11:37:01.307977+0000","last_unstale":"2024-03-24T11:39:36.012692+0000","last_undegraded":"2024-03-24T11:39:36.012692+0000","last_fullsized":"2024-03-24T11:39:36.012692+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:38:54.329204+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_clean_scrub_stamp":"2024-03-24T11:38:54.329204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:20.416911+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"434'11727","reported_seq":13659,"reported_epoch":434,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T11:39:38.536842+0000","last_change":"2024-03-24T11:39:29.522293+0000","last_active":"2024-03-24T11:39:38.536842+0000","last_peered":"2024-03-24T11:39:38.536842+0000","last_clean":"2024-03-24T11:39:27.511399+0000","last_became_active":"2024-03-24T11:39:29.456474+0000","last_became_peered":"2024-03-24T11:39:29.456474+0000","last_unstale":"2024-03-24T11:39:38.536842+0000","last_undegraded":"2024-03-24T11:39:38.536842+0000","last_fullsized":"2024-03-24T11:39:38.536842+0000","mapping_epoch":429,"log_start":"417'8254","ondisk_log_start":"417'8254","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9080","last_scrub_stamp":"2024-03-24T11:38:57.081681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.081681+0000","objects_scrubbed":5915,"log_size":3473,"log_dups_size":0,"ondisk_log_size":3473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209125376,"num_objects":3191,"num_object_clones":0,"num_object_copies":9573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2382,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":57409536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,7],"avail_no_missing":["1","0","7"],"object_location_counts":[{"shards":"0,1,7","objects":3191}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"388'9","reported_seq":1058,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.383300+0000","last_change":"2024-03-24T11:36:58.931284+0000","last_active":"2024-03-24T11:39:39.383300+0000","last_peered":"2024-03-24T11:39:39.383300+0000","last_clean":"2024-03-24T11:39:39.383300+0000","last_became_active":"2024-03-24T11:36:58.591546+0000","last_became_peered":"2024-03-24T11:36:58.591546+0000","last_unstale":"2024-03-24T11:39:39.383300+0000","last_undegraded":"2024-03-24T11:39:39.383300+0000","last_fullsized":"2024-03-24T11:39:39.383300+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:58.931185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:58.931185+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'42","reported_seq":535,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.007884+0000","last_change":"2024-03-24T11:39:12.053807+0000","last_active":"2024-03-24T11:39:36.007884+0000","last_peered":"2024-03-24T11:39:36.007884+0000","last_clean":"2024-03-24T11:39:36.007884+0000","last_became_active":"2024-03-24T11:36:58.573987+0000","last_became_peered":"2024-03-24T11:36:58.573987+0000","last_unstale":"2024-03-24T11:39:36.007884+0000","last_undegraded":"2024-03-24T11:39:36.007884+0000","last_fullsized":"2024-03-24T11:39:36.007884+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'42","last_scrub_stamp":"2024-03-24T11:39:12.053617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:12.053617+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:18.399042+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":849,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.018186+0000","last_change":"2024-03-24T11:39:28.592846+0000","last_active":"2024-03-24T11:39:36.018186+0000","last_peered":"2024-03-24T11:39:36.018186+0000","last_clean":"2024-03-24T11:39:36.018186+0000","last_became_active":"2024-03-24T11:39:28.576255+0000","last_became_peered":"2024-03-24T11:39:28.576255+0000","last_unstale":"2024-03-24T11:39:36.018186+0000","last_undegraded":"2024-03-24T11:39:36.018186+0000","last_fullsized":"2024-03-24T11:39:36.018186+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:09.054428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:09.054428+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:38.472256+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":446,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.010556+0000","last_change":"2024-03-24T11:39:28.572576+0000","last_active":"2024-03-24T11:39:36.010556+0000","last_peered":"2024-03-24T11:39:36.010556+0000","last_clean":"2024-03-24T11:39:36.010556+0000","last_became_active":"2024-03-24T11:39:28.572211+0000","last_became_peered":"2024-03-24T11:39:28.572211+0000","last_unstale":"2024-03-24T11:39:36.010556+0000","last_undegraded":"2024-03-24T11:39:36.010556+0000","last_fullsized":"2024-03-24T11:39:36.010556+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:10.119478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.119478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:36.297701+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1011,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381460+0000","last_change":"2024-03-24T11:39:31.390790+0000","last_active":"2024-03-24T11:39:39.381460+0000","last_peered":"2024-03-24T11:39:39.381460+0000","last_clean":"2024-03-24T11:39:39.381460+0000","last_became_active":"2024-03-24T11:39:31.390386+0000","last_became_peered":"2024-03-24T11:39:31.390386+0000","last_unstale":"2024-03-24T11:39:39.381460+0000","last_undegraded":"2024-03-24T11:39:39.381460+0000","last_fullsized":"2024-03-24T11:39:39.381460+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:42.045652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.045652+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"434'12121","reported_seq":12314,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:38.722747+0000","last_change":"2024-03-24T11:38:53.481972+0000","last_active":"2024-03-24T11:39:38.722747+0000","last_peered":"2024-03-24T11:39:38.722747+0000","last_clean":"2024-03-24T11:39:38.722747+0000","last_became_active":"2024-03-24T11:36:58.589952+0000","last_became_peered":"2024-03-24T11:36:58.589952+0000","last_unstale":"2024-03-24T11:39:38.722747+0000","last_undegraded":"2024-03-24T11:39:38.722747+0000","last_fullsized":"2024-03-24T11:39:38.722747+0000","mapping_epoch":387,"log_start":"415'7426","ondisk_log_start":"415'7426","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'9128","last_scrub_stamp":"2024-03-24T11:38:53.481477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.481477+0000","objects_scrubbed":6260,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.975145+0000","scrub_duration":2214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209846296,"num_objects":3203,"num_object_clones":0,"num_object_copies":9609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12121,"num_write_kb":490305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":1078,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.017200+0000","last_change":"2024-03-24T11:39:02.353239+0000","last_active":"2024-03-24T11:39:36.017200+0000","last_peered":"2024-03-24T11:39:36.017200+0000","last_clean":"2024-03-24T11:39:36.017200+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:39:36.017200+0000","last_undegraded":"2024-03-24T11:39:36.017200+0000","last_fullsized":"2024-03-24T11:39:36.017200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:39:02.353110+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.353110+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.696468+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'30","reported_seq":895,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998288+0000","last_change":"2024-03-24T11:38:53.578501+0000","last_active":"2024-03-24T11:39:35.998288+0000","last_peered":"2024-03-24T11:39:35.998288+0000","last_clean":"2024-03-24T11:39:35.998288+0000","last_became_active":"2024-03-24T11:36:58.597641+0000","last_became_peered":"2024-03-24T11:36:58.597641+0000","last_unstale":"2024-03-24T11:39:35.998288+0000","last_undegraded":"2024-03-24T11:39:35.998288+0000","last_fullsized":"2024-03-24T11:39:35.998288+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:53.578365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.578365+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:19.628891+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":839,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009348+0000","last_change":"2024-03-24T11:39:16.120140+0000","last_active":"2024-03-24T11:39:36.009348+0000","last_peered":"2024-03-24T11:39:36.009348+0000","last_clean":"2024-03-24T11:39:36.009348+0000","last_became_active":"2024-03-24T11:18:53.680712+0000","last_became_peered":"2024-03-24T11:18:53.680712+0000","last_unstale":"2024-03-24T11:39:36.009348+0000","last_undegraded":"2024-03-24T11:39:36.009348+0000","last_fullsized":"2024-03-24T11:39:36.009348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:16.120029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_clean_scrub_stamp":"2024-03-24T11:39:16.120029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:25.051058+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":810,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.014041+0000","last_change":"2024-03-24T11:39:01.453653+0000","last_active":"2024-03-24T11:39:36.014041+0000","last_peered":"2024-03-24T11:39:36.014041+0000","last_clean":"2024-03-24T11:39:36.014041+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:39:36.014041+0000","last_undegraded":"2024-03-24T11:39:36.014041+0000","last_fullsized":"2024-03-24T11:39:36.014041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:39:01.453536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.453536+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.228643+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1028,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.010652+0000","last_change":"2024-03-24T11:39:31.396058+0000","last_active":"2024-03-24T11:39:36.010652+0000","last_peered":"2024-03-24T11:39:36.010652+0000","last_clean":"2024-03-24T11:39:36.010652+0000","last_became_active":"2024-03-24T11:39:31.395565+0000","last_became_peered":"2024-03-24T11:39:31.395565+0000","last_unstale":"2024-03-24T11:39:36.010652+0000","last_undegraded":"2024-03-24T11:39:36.010652+0000","last_fullsized":"2024-03-24T11:39:36.010652+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:41.909070+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.909070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"434'11717","reported_seq":13928,"reported_epoch":434,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T11:39:38.528221+0000","last_change":"2024-03-24T11:39:29.534602+0000","last_active":"2024-03-24T11:39:38.528221+0000","last_peered":"2024-03-24T11:39:38.528221+0000","last_clean":"2024-03-24T11:39:27.519545+0000","last_became_active":"2024-03-24T11:39:29.456482+0000","last_became_peered":"2024-03-24T11:39:29.456482+0000","last_unstale":"2024-03-24T11:39:38.528221+0000","last_undegraded":"2024-03-24T11:39:38.528221+0000","last_fullsized":"2024-03-24T11:39:38.528221+0000","mapping_epoch":429,"log_start":"416'8211","ondisk_log_start":"416'8211","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":1899,"log_size":3506,"log_dups_size":0,"ondisk_log_size":3506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211091456,"num_objects":3221,"num_object_clones":0,"num_object_copies":9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2277,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11717,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":3221}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":836,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009739+0000","last_change":"2024-03-24T11:37:43.909192+0000","last_active":"2024-03-24T11:39:36.009739+0000","last_peered":"2024-03-24T11:39:36.009739+0000","last_clean":"2024-03-24T11:39:36.009739+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:39:36.009739+0000","last_undegraded":"2024-03-24T11:39:36.009739+0000","last_fullsized":"2024-03-24T11:39:36.009739+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.909062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.909062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'39","reported_seq":906,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.016832+0000","last_change":"2024-03-24T11:39:28.596486+0000","last_active":"2024-03-24T11:39:36.016832+0000","last_peered":"2024-03-24T11:39:36.016832+0000","last_clean":"2024-03-24T11:39:36.016832+0000","last_became_active":"2024-03-24T11:39:28.550731+0000","last_became_peered":"2024-03-24T11:39:28.550731+0000","last_unstale":"2024-03-24T11:39:36.016832+0000","last_undegraded":"2024-03-24T11:39:36.016832+0000","last_fullsized":"2024-03-24T11:39:36.016832+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'29","last_scrub_stamp":"2024-03-24T11:38:04.942715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.942715+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:38.295941+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":836,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.026059+0000","last_change":"2024-03-24T11:37:49.656241+0000","last_active":"2024-03-24T11:39:36.026059+0000","last_peered":"2024-03-24T11:39:36.026059+0000","last_clean":"2024-03-24T11:39:36.026059+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:39:36.026059+0000","last_undegraded":"2024-03-24T11:39:36.026059+0000","last_fullsized":"2024-03-24T11:39:36.026059+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:49.656104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:49.656104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":816,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998925+0000","last_change":"2024-03-24T11:37:45.747922+0000","last_active":"2024-03-24T11:39:35.998925+0000","last_peered":"2024-03-24T11:39:35.998925+0000","last_clean":"2024-03-24T11:39:35.998925+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:39:35.998925+0000","last_undegraded":"2024-03-24T11:39:35.998925+0000","last_fullsized":"2024-03-24T11:39:35.998925+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:45.747799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:45.747799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1021,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.011235+0000","last_change":"2024-03-24T11:39:14.270225+0000","last_active":"2024-03-24T11:39:36.011235+0000","last_peered":"2024-03-24T11:39:36.011235+0000","last_clean":"2024-03-24T11:39:36.011235+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:39:36.011235+0000","last_undegraded":"2024-03-24T11:39:36.011235+0000","last_fullsized":"2024-03-24T11:39:36.011235+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_clean_scrub_stamp":"2024-03-24T11:39:14.270084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:40.997167+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"435'12046","reported_seq":12162,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.603294+0000","last_change":"2024-03-24T11:36:51.535382+0000","last_active":"2024-03-24T11:39:39.603294+0000","last_peered":"2024-03-24T11:39:39.603294+0000","last_clean":"2024-03-24T11:39:39.603294+0000","last_became_active":"2024-03-24T11:36:51.534255+0000","last_became_peered":"2024-03-24T11:36:51.534255+0000","last_unstale":"2024-03-24T11:39:39.603294+0000","last_undegraded":"2024-03-24T11:39:39.603294+0000","last_fullsized":"2024-03-24T11:39:39.603294+0000","mapping_epoch":382,"log_start":"417'8503","ondisk_log_start":"417'8503","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208142336,"num_objects":3176,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":820,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009813+0000","last_change":"2024-03-24T11:39:14.156792+0000","last_active":"2024-03-24T11:39:36.009813+0000","last_peered":"2024-03-24T11:39:36.009813+0000","last_clean":"2024-03-24T11:39:36.009813+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:39:36.009813+0000","last_undegraded":"2024-03-24T11:39:36.009813+0000","last_fullsized":"2024-03-24T11:39:36.009813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:14.156684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:39:14.156684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:23.315132+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'36","reported_seq":387,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008620+0000","last_change":"2024-03-24T11:39:01.923217+0000","last_active":"2024-03-24T11:39:36.008620+0000","last_peered":"2024-03-24T11:39:36.008620+0000","last_clean":"2024-03-24T11:39:36.008620+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:39:36.008620+0000","last_undegraded":"2024-03-24T11:39:36.008620+0000","last_fullsized":"2024-03-24T11:39:36.008620+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'28","last_scrub_stamp":"2024-03-24T11:39:01.923083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.923083+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:20.126246+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1026,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.608040+0000","last_change":"2024-03-24T11:39:28.601193+0000","last_active":"2024-03-24T11:39:36.608040+0000","last_peered":"2024-03-24T11:39:36.608040+0000","last_clean":"2024-03-24T11:39:36.608040+0000","last_became_active":"2024-03-24T11:39:28.571981+0000","last_became_peered":"2024-03-24T11:39:28.571981+0000","last_unstale":"2024-03-24T11:39:36.608040+0000","last_undegraded":"2024-03-24T11:39:36.608040+0000","last_fullsized":"2024-03-24T11:39:36.608040+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:37:03.767151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:37:03.767151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":543,"num_read_kb":538,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":837,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.023277+0000","last_change":"2024-03-24T11:39:06.423843+0000","last_active":"2024-03-24T11:39:36.023277+0000","last_peered":"2024-03-24T11:39:36.023277+0000","last_clean":"2024-03-24T11:39:36.023277+0000","last_became_active":"2024-03-24T11:36:58.572807+0000","last_became_peered":"2024-03-24T11:36:58.572807+0000","last_unstale":"2024-03-24T11:39:36.023277+0000","last_undegraded":"2024-03-24T11:39:36.023277+0000","last_fullsized":"2024-03-24T11:39:36.023277+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:06.423513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:39:06.423513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:30.290729+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":824,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.024251+0000","last_change":"2024-03-24T11:38:05.656269+0000","last_active":"2024-03-24T11:39:36.024251+0000","last_peered":"2024-03-24T11:39:36.024251+0000","last_clean":"2024-03-24T11:39:36.024251+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:39:36.024251+0000","last_undegraded":"2024-03-24T11:39:36.024251+0000","last_fullsized":"2024-03-24T11:39:36.024251+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.656146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.656146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1024,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.002838+0000","last_change":"2024-03-24T11:37:01.308329+0000","last_active":"2024-03-24T11:39:36.002838+0000","last_peered":"2024-03-24T11:39:36.002838+0000","last_clean":"2024-03-24T11:39:36.002838+0000","last_became_active":"2024-03-24T11:37:01.307925+0000","last_became_peered":"2024-03-24T11:37:01.307925+0000","last_unstale":"2024-03-24T11:39:36.002838+0000","last_undegraded":"2024-03-24T11:39:36.002838+0000","last_fullsized":"2024-03-24T11:39:36.002838+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:49.592471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:49.592471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"435'11807","reported_seq":11923,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.591049+0000","last_change":"2024-03-24T11:36:51.533861+0000","last_active":"2024-03-24T11:39:39.591049+0000","last_peered":"2024-03-24T11:39:39.591049+0000","last_clean":"2024-03-24T11:39:39.591049+0000","last_became_active":"2024-03-24T11:36:51.533003+0000","last_became_peered":"2024-03-24T11:36:51.533003+0000","last_unstale":"2024-03-24T11:39:39.591049+0000","last_undegraded":"2024-03-24T11:39:39.591049+0000","last_fullsized":"2024-03-24T11:39:39.591049+0000","mapping_epoch":382,"log_start":"417'8271","ondisk_log_start":"417'8271","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205586432,"num_objects":3137,"num_object_clones":0,"num_object_copies":9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":478208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":446,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009877+0000","last_change":"2024-03-24T11:39:28.570349+0000","last_active":"2024-03-24T11:39:36.009877+0000","last_peered":"2024-03-24T11:39:36.009877+0000","last_clean":"2024-03-24T11:39:36.009877+0000","last_became_active":"2024-03-24T11:39:28.569858+0000","last_became_peered":"2024-03-24T11:39:28.569858+0000","last_unstale":"2024-03-24T11:39:36.009877+0000","last_undegraded":"2024-03-24T11:39:36.009877+0000","last_fullsized":"2024-03-24T11:39:36.009877+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.438380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.438380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"429'50","reported_seq":1203,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.013153+0000","last_change":"2024-03-24T11:39:28.601191+0000","last_active":"2024-03-24T11:39:36.013153+0000","last_peered":"2024-03-24T11:39:36.013153+0000","last_clean":"2024-03-24T11:39:36.013153+0000","last_became_active":"2024-03-24T11:39:28.564381+0000","last_became_peered":"2024-03-24T11:39:28.564381+0000","last_unstale":"2024-03-24T11:39:36.013153+0000","last_undegraded":"2024-03-24T11:39:36.013153+0000","last_fullsized":"2024-03-24T11:39:36.013153+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-03-24T11:37:52.944533+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.944533+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":817,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.388612+0000","last_change":"2024-03-24T11:39:28.585112+0000","last_active":"2024-03-24T11:39:39.388612+0000","last_peered":"2024-03-24T11:39:39.388612+0000","last_clean":"2024-03-24T11:39:39.388612+0000","last_became_active":"2024-03-24T11:39:28.584724+0000","last_became_peered":"2024-03-24T11:39:28.584724+0000","last_unstale":"2024-03-24T11:39:39.388612+0000","last_undegraded":"2024-03-24T11:39:39.388612+0000","last_fullsized":"2024-03-24T11:39:39.388612+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.978638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.978638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.014131+0000","last_change":"2024-03-24T11:39:31.397589+0000","last_active":"2024-03-24T11:39:36.014131+0000","last_peered":"2024-03-24T11:39:36.014131+0000","last_clean":"2024-03-24T11:39:36.014131+0000","last_became_active":"2024-03-24T11:39:31.397237+0000","last_became_peered":"2024-03-24T11:39:31.397237+0000","last_unstale":"2024-03-24T11:39:36.014131+0000","last_undegraded":"2024-03-24T11:39:36.014131+0000","last_fullsized":"2024-03-24T11:39:36.014131+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:39:03.394244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:39:03.394244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:11.252029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":830,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009439+0000","last_change":"2024-03-24T11:38:02.291361+0000","last_active":"2024-03-24T11:39:36.009439+0000","last_peered":"2024-03-24T11:39:36.009439+0000","last_clean":"2024-03-24T11:39:36.009439+0000","last_became_active":"2024-03-24T11:36:58.586913+0000","last_became_peered":"2024-03-24T11:36:58.586913+0000","last_unstale":"2024-03-24T11:39:36.009439+0000","last_undegraded":"2024-03-24T11:39:36.009439+0000","last_fullsized":"2024-03-24T11:39:36.009439+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:02.291240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:38:02.291240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"435'11927","reported_seq":12043,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.575289+0000","last_change":"2024-03-24T11:36:51.530980+0000","last_active":"2024-03-24T11:39:39.575289+0000","last_peered":"2024-03-24T11:39:39.575289+0000","last_clean":"2024-03-24T11:39:39.575289+0000","last_became_active":"2024-03-24T11:36:51.530664+0000","last_became_peered":"2024-03-24T11:36:51.530664+0000","last_unstale":"2024-03-24T11:39:39.575289+0000","last_undegraded":"2024-03-24T11:39:39.575289+0000","last_fullsized":"2024-03-24T11:39:39.575289+0000","mapping_epoch":382,"log_start":"417'8379","ondisk_log_start":"417'8379","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205324288,"num_objects":3133,"num_object_clones":0,"num_object_copies":9399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11927,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":850,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.005814+0000","last_change":"2024-03-24T11:39:28.575114+0000","last_active":"2024-03-24T11:39:36.005814+0000","last_peered":"2024-03-24T11:39:36.005814+0000","last_clean":"2024-03-24T11:39:36.005814+0000","last_became_active":"2024-03-24T11:39:28.574632+0000","last_became_peered":"2024-03-24T11:39:28.574632+0000","last_unstale":"2024-03-24T11:39:36.005814+0000","last_undegraded":"2024-03-24T11:39:36.005814+0000","last_fullsized":"2024-03-24T11:39:36.005814+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:13.234123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:13.234123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:19.803160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"421'21","reported_seq":854,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.011581+0000","last_change":"2024-03-24T11:39:01.291942+0000","last_active":"2024-03-24T11:39:36.011581+0000","last_peered":"2024-03-24T11:39:36.011581+0000","last_clean":"2024-03-24T11:39:36.011581+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:39:36.011581+0000","last_undegraded":"2024-03-24T11:39:36.011581+0000","last_fullsized":"2024-03-24T11:39:36.011581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'19","last_scrub_stamp":"2024-03-24T11:39:01.291795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.291795+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.103347+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":853,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.005824+0000","last_change":"2024-03-24T11:37:59.474830+0000","last_active":"2024-03-24T11:39:36.005824+0000","last_peered":"2024-03-24T11:39:36.005824+0000","last_clean":"2024-03-24T11:39:36.005824+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:39:36.005824+0000","last_undegraded":"2024-03-24T11:39:36.005824+0000","last_fullsized":"2024-03-24T11:39:36.005824+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:59.474714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:37:59.474714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:38.369071+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":837,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.995666+0000","last_change":"2024-03-24T11:37:53.683206+0000","last_active":"2024-03-24T11:39:35.995666+0000","last_peered":"2024-03-24T11:39:35.995666+0000","last_clean":"2024-03-24T11:39:35.995666+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:39:35.995666+0000","last_undegraded":"2024-03-24T11:39:35.995666+0000","last_fullsized":"2024-03-24T11:39:35.995666+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:53.683090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:53.683090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":446,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.013181+0000","last_change":"2024-03-24T11:39:28.562110+0000","last_active":"2024-03-24T11:39:36.013181+0000","last_peered":"2024-03-24T11:39:36.013181+0000","last_clean":"2024-03-24T11:39:36.013181+0000","last_became_active":"2024-03-24T11:39:28.561579+0000","last_became_peered":"2024-03-24T11:39:28.561579+0000","last_unstale":"2024-03-24T11:39:36.013181+0000","last_undegraded":"2024-03-24T11:39:36.013181+0000","last_fullsized":"2024-03-24T11:39:36.013181+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:55.094992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:38:55.094992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.216275+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":800,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.028590+0000","last_change":"2024-03-24T11:39:30.376213+0000","last_active":"2024-03-24T11:39:36.028590+0000","last_peered":"2024-03-24T11:39:36.028590+0000","last_clean":"2024-03-24T11:39:36.028590+0000","last_became_active":"2024-03-24T11:39:30.375769+0000","last_became_peered":"2024-03-24T11:39:30.375769+0000","last_unstale":"2024-03-24T11:39:36.028590+0000","last_undegraded":"2024-03-24T11:39:36.028590+0000","last_fullsized":"2024-03-24T11:39:36.028590+0000","mapping_epoch":430,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:00.701398+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:36:00.701398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:38.295803+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"435'12022","reported_seq":12138,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.642304+0000","last_change":"2024-03-24T11:36:51.532092+0000","last_active":"2024-03-24T11:39:39.642304+0000","last_peered":"2024-03-24T11:39:39.642304+0000","last_clean":"2024-03-24T11:39:39.642304+0000","last_became_active":"2024-03-24T11:36:51.531785+0000","last_became_peered":"2024-03-24T11:36:51.531785+0000","last_unstale":"2024-03-24T11:39:39.642304+0000","last_undegraded":"2024-03-24T11:39:39.642304+0000","last_fullsized":"2024-03-24T11:39:39.642304+0000","mapping_epoch":382,"log_start":"417'8459","ondisk_log_start":"417'8459","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:36:50.511060+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201457664,"num_objects":3074,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":483072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":818,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.014907+0000","last_change":"2024-03-24T11:37:51.788499+0000","last_active":"2024-03-24T11:39:36.014907+0000","last_peered":"2024-03-24T11:39:36.014907+0000","last_clean":"2024-03-24T11:39:36.014907+0000","last_became_active":"2024-03-24T11:36:58.594154+0000","last_became_peered":"2024-03-24T11:36:58.594154+0000","last_unstale":"2024-03-24T11:39:36.014907+0000","last_undegraded":"2024-03-24T11:39:36.014907+0000","last_fullsized":"2024-03-24T11:39:36.014907+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.788360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.788360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:38.295838+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"421'53","reported_seq":1185,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.027070+0000","last_change":"2024-03-24T11:37:44.707841+0000","last_active":"2024-03-24T11:39:36.027070+0000","last_peered":"2024-03-24T11:39:36.027070+0000","last_clean":"2024-03-24T11:39:36.027070+0000","last_became_active":"2024-03-24T11:37:33.888447+0000","last_became_peered":"2024-03-24T11:37:33.888447+0000","last_unstale":"2024-03-24T11:39:36.027070+0000","last_undegraded":"2024-03-24T11:39:36.027070+0000","last_fullsized":"2024-03-24T11:39:36.027070+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'41","last_scrub_stamp":"2024-03-24T11:37:44.707693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.707693+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":113,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.016517+0000","last_change":"2024-03-24T11:38:01.964312+0000","last_active":"2024-03-24T11:39:36.016517+0000","last_peered":"2024-03-24T11:39:36.016517+0000","last_clean":"2024-03-24T11:39:36.016517+0000","last_became_active":"2024-03-24T11:36:58.591467+0000","last_became_peered":"2024-03-24T11:36:58.591467+0000","last_unstale":"2024-03-24T11:39:36.016517+0000","last_undegraded":"2024-03-24T11:39:36.016517+0000","last_fullsized":"2024-03-24T11:39:36.016517+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.964189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:38.295867+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":839,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008087+0000","last_change":"2024-03-24T11:39:05.008974+0000","last_active":"2024-03-24T11:39:36.008087+0000","last_peered":"2024-03-24T11:39:36.008087+0000","last_clean":"2024-03-24T11:39:36.008087+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:39:36.008087+0000","last_undegraded":"2024-03-24T11:39:36.008087+0000","last_fullsized":"2024-03-24T11:39:36.008087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:39:05.008874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:39:05.008874+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:06.255132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":805,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.386281+0000","last_change":"2024-03-24T11:39:28.580748+0000","last_active":"2024-03-24T11:39:39.386281+0000","last_peered":"2024-03-24T11:39:39.386281+0000","last_clean":"2024-03-24T11:39:39.386281+0000","last_became_active":"2024-03-24T11:39:28.576107+0000","last_became_peered":"2024-03-24T11:39:28.576107+0000","last_unstale":"2024-03-24T11:39:39.386281+0000","last_undegraded":"2024-03-24T11:39:39.386281+0000","last_fullsized":"2024-03-24T11:39:39.386281+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:02.217239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.217239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:27.167797+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1007,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.007857+0000","last_change":"2024-03-24T11:37:55.105164+0000","last_active":"2024-03-24T11:39:36.007857+0000","last_peered":"2024-03-24T11:39:36.007857+0000","last_clean":"2024-03-24T11:39:36.007857+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:39:36.007857+0000","last_undegraded":"2024-03-24T11:39:36.007857+0000","last_fullsized":"2024-03-24T11:39:36.007857+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:55.105007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.105007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"434'11810","reported_seq":13595,"reported_epoch":434,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T11:39:38.616956+0000","last_change":"2024-03-24T11:39:29.525261+0000","last_active":"2024-03-24T11:39:38.616956+0000","last_peered":"2024-03-24T11:39:38.616956+0000","last_clean":"2024-03-24T11:39:27.530503+0000","last_became_active":"2024-03-24T11:39:29.459760+0000","last_became_peered":"2024-03-24T11:39:29.459760+0000","last_unstale":"2024-03-24T11:39:38.616956+0000","last_undegraded":"2024-03-24T11:39:38.616956+0000","last_fullsized":"2024-03-24T11:39:38.616956+0000","mapping_epoch":429,"log_start":"415'7860","ondisk_log_start":"415'7860","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'9669","last_scrub_stamp":"2024-03-24T11:39:04.420545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:39:04.420545+0000","objects_scrubbed":5289,"log_size":3950,"log_dups_size":0,"ondisk_log_size":3950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2368,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":476672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":52494336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,7],"avail_no_missing":["6","2","7"],"object_location_counts":[{"shards":"2,6,7","objects":3086}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":874,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.017102+0000","last_change":"2024-03-24T11:39:12.419766+0000","last_active":"2024-03-24T11:39:36.017102+0000","last_peered":"2024-03-24T11:39:36.017102+0000","last_clean":"2024-03-24T11:39:36.017102+0000","last_became_active":"2024-03-24T11:36:58.590879+0000","last_became_peered":"2024-03-24T11:36:58.590879+0000","last_unstale":"2024-03-24T11:39:36.017102+0000","last_undegraded":"2024-03-24T11:39:36.017102+0000","last_fullsized":"2024-03-24T11:39:36.017102+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:39:12.418838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:12.418838+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:29.032656+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":834,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008282+0000","last_change":"2024-03-24T11:38:05.022910+0000","last_active":"2024-03-24T11:39:36.008282+0000","last_peered":"2024-03-24T11:39:36.008282+0000","last_clean":"2024-03-24T11:39:36.008282+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:39:36.008282+0000","last_undegraded":"2024-03-24T11:39:36.008282+0000","last_fullsized":"2024-03-24T11:39:36.008282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:05.022785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.022785+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":117,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.389054+0000","last_change":"2024-03-24T11:39:01.186035+0000","last_active":"2024-03-24T11:39:39.389054+0000","last_peered":"2024-03-24T11:39:39.389054+0000","last_clean":"2024-03-24T11:39:39.389054+0000","last_became_active":"2024-03-24T11:36:58.576200+0000","last_became_peered":"2024-03-24T11:36:58.576200+0000","last_unstale":"2024-03-24T11:39:39.389054+0000","last_undegraded":"2024-03-24T11:39:39.389054+0000","last_fullsized":"2024-03-24T11:39:39.389054+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:01.185721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.185721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:27.043011+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":814,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.023613+0000","last_change":"2024-03-24T11:39:28.568118+0000","last_active":"2024-03-24T11:39:36.023613+0000","last_peered":"2024-03-24T11:39:36.023613+0000","last_clean":"2024-03-24T11:39:36.023613+0000","last_became_active":"2024-03-24T11:39:28.567784+0000","last_became_peered":"2024-03-24T11:39:28.567784+0000","last_unstale":"2024-03-24T11:39:36.023613+0000","last_undegraded":"2024-03-24T11:39:36.023613+0000","last_fullsized":"2024-03-24T11:39:36.023613+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.856841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.856841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"434'11918","reported_seq":14807,"reported_epoch":434,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T11:39:38.711239+0000","last_change":"2024-03-24T11:39:29.511975+0000","last_active":"2024-03-24T11:39:38.711239+0000","last_peered":"2024-03-24T11:39:38.711239+0000","last_clean":"2024-03-24T11:39:27.513245+0000","last_became_active":"2024-03-24T11:39:29.474037+0000","last_became_peered":"2024-03-24T11:39:29.474037+0000","last_unstale":"2024-03-24T11:39:38.711239+0000","last_undegraded":"2024-03-24T11:39:38.711239+0000","last_fullsized":"2024-03-24T11:39:38.711239+0000","mapping_epoch":429,"log_start":"416'8286","ondisk_log_start":"416'8286","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9589","last_scrub_stamp":"2024-03-24T11:39:01.819797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.819797+0000","objects_scrubbed":5600,"log_size":3632,"log_dups_size":0,"ondisk_log_size":3632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212992000,"num_objects":3250,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1352,"num_bytes_recovered":88604672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":3250}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":946,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009736+0000","last_change":"2024-03-24T11:39:28.589850+0000","last_active":"2024-03-24T11:39:36.009736+0000","last_peered":"2024-03-24T11:39:36.009736+0000","last_clean":"2024-03-24T11:39:36.009736+0000","last_became_active":"2024-03-24T11:39:28.547142+0000","last_became_peered":"2024-03-24T11:39:28.547142+0000","last_unstale":"2024-03-24T11:39:36.009736+0000","last_undegraded":"2024-03-24T11:39:36.009736+0000","last_fullsized":"2024-03-24T11:39:36.009736+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'44","last_scrub_stamp":"2024-03-24T11:37:34.852224+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:37:34.852224+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":438,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009755+0000","last_change":"2024-03-24T11:39:28.549557+0000","last_active":"2024-03-24T11:39:36.009755+0000","last_peered":"2024-03-24T11:39:36.009755+0000","last_clean":"2024-03-24T11:39:36.009755+0000","last_became_active":"2024-03-24T11:39:28.549238+0000","last_became_peered":"2024-03-24T11:39:28.549238+0000","last_unstale":"2024-03-24T11:39:36.009755+0000","last_undegraded":"2024-03-24T11:39:36.009755+0000","last_fullsized":"2024-03-24T11:39:36.009755+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:46.876805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:46.876805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:13.806074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":844,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.006303+0000","last_change":"2024-03-24T11:39:06.306923+0000","last_active":"2024-03-24T11:39:36.006303+0000","last_peered":"2024-03-24T11:39:36.006303+0000","last_clean":"2024-03-24T11:39:36.006303+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:39:36.006303+0000","last_undegraded":"2024-03-24T11:39:36.006303+0000","last_fullsized":"2024-03-24T11:39:36.006303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:06.306802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:39:06.306802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.729439+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":140,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.382601+0000","last_change":"2024-03-24T11:39:28.587339+0000","last_active":"2024-03-24T11:39:39.382601+0000","last_peered":"2024-03-24T11:39:39.382601+0000","last_clean":"2024-03-24T11:39:39.382601+0000","last_became_active":"2024-03-24T11:39:28.575619+0000","last_became_peered":"2024-03-24T11:39:28.575619+0000","last_unstale":"2024-03-24T11:39:39.382601+0000","last_undegraded":"2024-03-24T11:39:39.382601+0000","last_fullsized":"2024-03-24T11:39:39.382601+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.123323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"434'11909","reported_seq":12090,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:38.627203+0000","last_change":"2024-03-24T11:39:01.832083+0000","last_active":"2024-03-24T11:39:38.627203+0000","last_peered":"2024-03-24T11:39:38.627203+0000","last_clean":"2024-03-24T11:39:38.627203+0000","last_became_active":"2024-03-24T11:36:51.539372+0000","last_became_peered":"2024-03-24T11:36:51.539372+0000","last_unstale":"2024-03-24T11:39:38.627203+0000","last_undegraded":"2024-03-24T11:39:38.627203+0000","last_fullsized":"2024-03-24T11:39:38.627203+0000","mapping_epoch":382,"log_start":"415'7954","ondisk_log_start":"415'7954","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9621","last_scrub_stamp":"2024-03-24T11:39:01.831363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.831363+0000","objects_scrubbed":5575,"log_size":3955,"log_dups_size":0,"ondisk_log_size":3955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.804535+0000","scrub_duration":2492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209911808,"num_objects":3203,"num_object_clones":0,"num_object_copies":9609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11909,"num_write_kb":483584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":131,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.014342+0000","last_change":"2024-03-24T11:38:11.115502+0000","last_active":"2024-03-24T11:39:36.014342+0000","last_peered":"2024-03-24T11:39:36.014342+0000","last_clean":"2024-03-24T11:39:36.014342+0000","last_became_active":"2024-03-24T11:36:42.470376+0000","last_became_peered":"2024-03-24T11:36:42.470376+0000","last_unstale":"2024-03-24T11:39:36.014342+0000","last_undegraded":"2024-03-24T11:39:36.014342+0000","last_fullsized":"2024-03-24T11:39:36.014342+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:11.115390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.115390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:37.708874+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"421'24","reported_seq":509,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998639+0000","last_change":"2024-03-24T11:38:52.571057+0000","last_active":"2024-03-24T11:39:35.998639+0000","last_peered":"2024-03-24T11:39:35.998639+0000","last_clean":"2024-03-24T11:39:35.998639+0000","last_became_active":"2024-03-24T11:36:58.593181+0000","last_became_peered":"2024-03-24T11:36:58.593181+0000","last_unstale":"2024-03-24T11:39:35.998639+0000","last_undegraded":"2024-03-24T11:39:35.998639+0000","last_fullsized":"2024-03-24T11:39:35.998639+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:38:52.570919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.570919+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.066756+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":121,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.006415+0000","last_change":"2024-03-24T11:39:28.579352+0000","last_active":"2024-03-24T11:39:36.006415+0000","last_peered":"2024-03-24T11:39:36.006415+0000","last_clean":"2024-03-24T11:39:36.006415+0000","last_became_active":"2024-03-24T11:39:28.578963+0000","last_became_peered":"2024-03-24T11:39:28.578963+0000","last_unstale":"2024-03-24T11:39:36.006415+0000","last_undegraded":"2024-03-24T11:39:36.006415+0000","last_fullsized":"2024-03-24T11:39:36.006415+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:47.673879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:37:47.673879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":791,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381626+0000","last_change":"2024-03-24T11:39:28.584131+0000","last_active":"2024-03-24T11:39:39.381626+0000","last_peered":"2024-03-24T11:39:39.381626+0000","last_clean":"2024-03-24T11:39:39.381626+0000","last_became_active":"2024-03-24T11:39:28.581732+0000","last_became_peered":"2024-03-24T11:39:28.581732+0000","last_unstale":"2024-03-24T11:39:39.381626+0000","last_undegraded":"2024-03-24T11:39:39.381626+0000","last_fullsized":"2024-03-24T11:39:39.381626+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:42.746752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:33:42.746752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"434'11807","reported_seq":11995,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:38.701486+0000","last_change":"2024-03-24T11:38:57.445535+0000","last_active":"2024-03-24T11:39:38.701486+0000","last_peered":"2024-03-24T11:39:38.701486+0000","last_clean":"2024-03-24T11:39:38.701486+0000","last_became_active":"2024-03-24T11:36:58.587355+0000","last_became_peered":"2024-03-24T11:36:58.587355+0000","last_unstale":"2024-03-24T11:39:38.701486+0000","last_undegraded":"2024-03-24T11:39:38.701486+0000","last_fullsized":"2024-03-24T11:39:38.701486+0000","mapping_epoch":387,"log_start":"416'8197","ondisk_log_start":"416'8197","created":382,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9159","last_scrub_stamp":"2024-03-24T11:38:57.445359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:50.511060+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.445359+0000","objects_scrubbed":5752,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:26.730608+0000","scrub_duration":1611,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200343552,"num_objects":3057,"num_object_clones":0,"num_object_copies":9171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11807,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"429'12","reported_seq":595,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009443+0000","last_change":"2024-03-24T11:39:28.594217+0000","last_active":"2024-03-24T11:39:36.009443+0000","last_peered":"2024-03-24T11:39:36.009443+0000","last_clean":"2024-03-24T11:39:36.009443+0000","last_became_active":"2024-03-24T11:39:28.577500+0000","last_became_peered":"2024-03-24T11:39:28.577500+0000","last_unstale":"2024-03-24T11:39:36.009443+0000","last_undegraded":"2024-03-24T11:39:36.009443+0000","last_fullsized":"2024-03-24T11:39:36.009443+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'11","last_scrub_stamp":"2024-03-24T11:37:58.943622+0000","last_deep_scrub":"357'10","last_deep_scrub_stamp":"2024-03-24T11:35:53.386928+0000","last_clean_scrub_stamp":"2024-03-24T11:37:58.943622+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"421'46","reported_seq":914,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.007711+0000","last_change":"2024-03-24T11:39:28.598972+0000","last_active":"2024-03-24T11:39:36.007711+0000","last_peered":"2024-03-24T11:39:36.007711+0000","last_clean":"2024-03-24T11:39:36.007711+0000","last_became_active":"2024-03-24T11:39:28.563427+0000","last_became_peered":"2024-03-24T11:39:28.563427+0000","last_unstale":"2024-03-24T11:39:36.007711+0000","last_undegraded":"2024-03-24T11:39:36.007711+0000","last_fullsized":"2024-03-24T11:39:36.007711+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:38:00.140160+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:38:00.140160+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":333,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.016373+0000","last_change":"2024-03-24T11:39:28.602801+0000","last_active":"2024-03-24T11:39:36.016373+0000","last_peered":"2024-03-24T11:39:36.016373+0000","last_clean":"2024-03-24T11:39:36.016373+0000","last_became_active":"2024-03-24T11:39:28.582617+0000","last_became_peered":"2024-03-24T11:39:28.582617+0000","last_unstale":"2024-03-24T11:39:36.016373+0000","last_undegraded":"2024-03-24T11:39:36.016373+0000","last_fullsized":"2024-03-24T11:39:36.016373+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:37:37.707835+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.707835+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":816,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008916+0000","last_change":"2024-03-24T11:38:04.034506+0000","last_active":"2024-03-24T11:39:36.008916+0000","last_peered":"2024-03-24T11:39:36.008916+0000","last_clean":"2024-03-24T11:39:36.008916+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:39:36.008916+0000","last_undegraded":"2024-03-24T11:39:36.008916+0000","last_fullsized":"2024-03-24T11:39:36.008916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:04.034411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.034411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":836,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381254+0000","last_change":"2024-03-24T11:39:28.607055+0000","last_active":"2024-03-24T11:39:39.381254+0000","last_peered":"2024-03-24T11:39:39.381254+0000","last_clean":"2024-03-24T11:39:39.381254+0000","last_became_active":"2024-03-24T11:39:28.583847+0000","last_became_peered":"2024-03-24T11:39:28.583847+0000","last_unstale":"2024-03-24T11:39:39.381254+0000","last_undegraded":"2024-03-24T11:39:39.381254+0000","last_fullsized":"2024-03-24T11:39:39.381254+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:35:53.797708+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:35:53.797708+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":843,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008938+0000","last_change":"2024-03-24T11:38:00.904773+0000","last_active":"2024-03-24T11:39:36.008938+0000","last_peered":"2024-03-24T11:39:36.008938+0000","last_clean":"2024-03-24T11:39:36.008938+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:39:36.008938+0000","last_undegraded":"2024-03-24T11:39:36.008938+0000","last_fullsized":"2024-03-24T11:39:36.008938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:00.904671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:00.904671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":811,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008652+0000","last_change":"2024-03-24T11:37:37.114801+0000","last_active":"2024-03-24T11:39:36.008652+0000","last_peered":"2024-03-24T11:39:36.008652+0000","last_clean":"2024-03-24T11:39:36.008652+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:39:36.008652+0000","last_undegraded":"2024-03-24T11:39:36.008652+0000","last_fullsized":"2024-03-24T11:39:36.008652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:37.114698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:37.114698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":451,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.013299+0000","last_change":"2024-03-24T11:39:28.548117+0000","last_active":"2024-03-24T11:39:36.013299+0000","last_peered":"2024-03-24T11:39:36.013299+0000","last_clean":"2024-03-24T11:39:36.013299+0000","last_became_active":"2024-03-24T11:39:28.547484+0000","last_became_peered":"2024-03-24T11:39:28.547484+0000","last_unstale":"2024-03-24T11:39:36.013299+0000","last_undegraded":"2024-03-24T11:39:36.013299+0000","last_fullsized":"2024-03-24T11:39:36.013299+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:54.040131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:32.919963+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.040131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"421'25","reported_seq":877,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.018360+0000","last_change":"2024-03-24T11:39:28.590958+0000","last_active":"2024-03-24T11:39:36.018360+0000","last_peered":"2024-03-24T11:39:36.018360+0000","last_clean":"2024-03-24T11:39:36.018360+0000","last_became_active":"2024-03-24T11:39:28.564216+0000","last_became_peered":"2024-03-24T11:39:28.564216+0000","last_unstale":"2024-03-24T11:39:36.018360+0000","last_undegraded":"2024-03-24T11:39:36.018360+0000","last_fullsized":"2024-03-24T11:39:36.018360+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:38:57.280974+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.280974+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.297255+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":121,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009556+0000","last_change":"2024-03-24T11:39:28.577780+0000","last_active":"2024-03-24T11:39:36.009556+0000","last_peered":"2024-03-24T11:39:36.009556+0000","last_clean":"2024-03-24T11:39:36.009556+0000","last_became_active":"2024-03-24T11:39:28.577340+0000","last_became_peered":"2024-03-24T11:39:28.577340+0000","last_unstale":"2024-03-24T11:39:36.009556+0000","last_undegraded":"2024-03-24T11:39:36.009556+0000","last_fullsized":"2024-03-24T11:39:36.009556+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:42.909043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.909043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":794,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381158+0000","last_change":"2024-03-24T11:39:28.583583+0000","last_active":"2024-03-24T11:39:39.381158+0000","last_peered":"2024-03-24T11:39:39.381158+0000","last_clean":"2024-03-24T11:39:39.381158+0000","last_became_active":"2024-03-24T11:39:28.581534+0000","last_became_peered":"2024-03-24T11:39:28.581534+0000","last_unstale":"2024-03-24T11:39:39.381158+0000","last_undegraded":"2024-03-24T11:39:39.381158+0000","last_fullsized":"2024-03-24T11:39:39.381158+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:33:32.456539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:33:32.456539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":843,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009779+0000","last_change":"2024-03-24T11:39:28.570608+0000","last_active":"2024-03-24T11:39:36.009779+0000","last_peered":"2024-03-24T11:39:36.009779+0000","last_clean":"2024-03-24T11:39:36.009779+0000","last_became_active":"2024-03-24T11:39:28.570115+0000","last_became_peered":"2024-03-24T11:39:28.570115+0000","last_unstale":"2024-03-24T11:39:36.009779+0000","last_undegraded":"2024-03-24T11:39:36.009779+0000","last_fullsized":"2024-03-24T11:39:36.009779+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:02.355825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.355825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.572896+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":889,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009809+0000","last_change":"2024-03-24T11:39:28.551439+0000","last_active":"2024-03-24T11:39:36.009809+0000","last_peered":"2024-03-24T11:39:36.009809+0000","last_clean":"2024-03-24T11:39:36.009809+0000","last_became_active":"2024-03-24T11:39:28.544005+0000","last_became_peered":"2024-03-24T11:39:28.544005+0000","last_unstale":"2024-03-24T11:39:36.009809+0000","last_undegraded":"2024-03-24T11:39:36.009809+0000","last_fullsized":"2024-03-24T11:39:36.009809+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:11.883396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.883396+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":833,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.028480+0000","last_change":"2024-03-24T11:39:28.575492+0000","last_active":"2024-03-24T11:39:36.028480+0000","last_peered":"2024-03-24T11:39:36.028480+0000","last_clean":"2024-03-24T11:39:36.028480+0000","last_became_active":"2024-03-24T11:39:28.575151+0000","last_became_peered":"2024-03-24T11:39:28.575151+0000","last_unstale":"2024-03-24T11:39:36.028480+0000","last_undegraded":"2024-03-24T11:39:36.028480+0000","last_fullsized":"2024-03-24T11:39:36.028480+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:07.446672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:55.793540+0000","last_clean_scrub_stamp":"2024-03-24T11:39:07.446672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:35.089393+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"429'9","reported_seq":1064,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.382115+0000","last_change":"2024-03-24T11:39:28.596125+0000","last_active":"2024-03-24T11:39:39.382115+0000","last_peered":"2024-03-24T11:39:39.382115+0000","last_clean":"2024-03-24T11:39:39.382115+0000","last_became_active":"2024-03-24T11:39:28.581602+0000","last_became_peered":"2024-03-24T11:39:28.581602+0000","last_unstale":"2024-03-24T11:39:39.382115+0000","last_undegraded":"2024-03-24T11:39:39.382115+0000","last_fullsized":"2024-03-24T11:39:39.382115+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:54.050291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:54.050291+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":121,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.011472+0000","last_change":"2024-03-24T11:38:50.540685+0000","last_active":"2024-03-24T11:39:36.011472+0000","last_peered":"2024-03-24T11:39:36.011472+0000","last_clean":"2024-03-24T11:39:36.011472+0000","last_became_active":"2024-03-24T11:36:58.580967+0000","last_became_peered":"2024-03-24T11:36:58.580967+0000","last_unstale":"2024-03-24T11:39:36.011472+0000","last_undegraded":"2024-03-24T11:39:36.011472+0000","last_fullsized":"2024-03-24T11:39:36.011472+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:50.540148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:38:50.540148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:10.416657+0000","scrub_duration":289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":855,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381958+0000","last_change":"2024-03-24T11:39:28.602629+0000","last_active":"2024-03-24T11:39:39.381958+0000","last_peered":"2024-03-24T11:39:39.381958+0000","last_clean":"2024-03-24T11:39:39.381958+0000","last_became_active":"2024-03-24T11:39:28.577653+0000","last_became_peered":"2024-03-24T11:39:28.577653+0000","last_unstale":"2024-03-24T11:39:39.381958+0000","last_undegraded":"2024-03-24T11:39:39.381958+0000","last_fullsized":"2024-03-24T11:39:39.381958+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:33:19.229920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:19.229920+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":121,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.006224+0000","last_change":"2024-03-24T11:39:28.567090+0000","last_active":"2024-03-24T11:39:36.006224+0000","last_peered":"2024-03-24T11:39:36.006224+0000","last_clean":"2024-03-24T11:39:36.006224+0000","last_became_active":"2024-03-24T11:39:28.566772+0000","last_became_peered":"2024-03-24T11:39:28.566772+0000","last_unstale":"2024-03-24T11:39:36.006224+0000","last_undegraded":"2024-03-24T11:39:36.006224+0000","last_fullsized":"2024-03-24T11:39:36.006224+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.220270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.220270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:25.681203+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":1076,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009208+0000","last_change":"2024-03-24T11:38:08.961040+0000","last_active":"2024-03-24T11:39:36.009208+0000","last_peered":"2024-03-24T11:39:36.009208+0000","last_clean":"2024-03-24T11:39:36.009208+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:39:36.009208+0000","last_undegraded":"2024-03-24T11:39:36.009208+0000","last_fullsized":"2024-03-24T11:39:36.009208+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:38:08.960909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.960909+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":813,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.018036+0000","last_change":"2024-03-24T11:39:28.561627+0000","last_active":"2024-03-24T11:39:36.018036+0000","last_peered":"2024-03-24T11:39:36.018036+0000","last_clean":"2024-03-24T11:39:36.018036+0000","last_became_active":"2024-03-24T11:39:28.561292+0000","last_became_peered":"2024-03-24T11:39:28.561292+0000","last_unstale":"2024-03-24T11:39:36.018036+0000","last_undegraded":"2024-03-24T11:39:36.018036+0000","last_fullsized":"2024-03-24T11:39:36.018036+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:09.604834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:09.604834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:28.087433+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'32","reported_seq":524,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.010309+0000","last_change":"2024-03-24T11:39:13.314319+0000","last_active":"2024-03-24T11:39:36.010309+0000","last_peered":"2024-03-24T11:39:36.010309+0000","last_clean":"2024-03-24T11:39:36.010309+0000","last_became_active":"2024-03-24T11:36:58.586291+0000","last_became_peered":"2024-03-24T11:36:58.586291+0000","last_unstale":"2024-03-24T11:39:36.010309+0000","last_undegraded":"2024-03-24T11:39:36.010309+0000","last_fullsized":"2024-03-24T11:39:36.010309+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:39:13.314202+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:39:13.314202+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:38.488654+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":119,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.010450+0000","last_change":"2024-03-24T11:39:28.564092+0000","last_active":"2024-03-24T11:39:36.010450+0000","last_peered":"2024-03-24T11:39:36.010450+0000","last_clean":"2024-03-24T11:39:36.010450+0000","last_became_active":"2024-03-24T11:39:28.563739+0000","last_became_peered":"2024-03-24T11:39:28.563739+0000","last_unstale":"2024-03-24T11:39:36.010450+0000","last_undegraded":"2024-03-24T11:39:36.010450+0000","last_fullsized":"2024-03-24T11:39:36.010450+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.070123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.070123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:27.938545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":827,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998208+0000","last_change":"2024-03-24T11:37:34.760552+0000","last_active":"2024-03-24T11:39:35.998208+0000","last_peered":"2024-03-24T11:39:35.998208+0000","last_clean":"2024-03-24T11:39:35.998208+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:39:35.998208+0000","last_undegraded":"2024-03-24T11:39:35.998208+0000","last_fullsized":"2024-03-24T11:39:35.998208+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:34.760436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:37:34.760436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":792,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.390028+0000","last_change":"2024-03-24T11:37:40.087780+0000","last_active":"2024-03-24T11:39:39.390028+0000","last_peered":"2024-03-24T11:39:39.390028+0000","last_clean":"2024-03-24T11:39:39.390028+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:39:39.390028+0000","last_undegraded":"2024-03-24T11:39:39.390028+0000","last_fullsized":"2024-03-24T11:39:39.390028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.086223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.086223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":857,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998380+0000","last_change":"2024-03-24T11:37:55.694778+0000","last_active":"2024-03-24T11:39:35.998380+0000","last_peered":"2024-03-24T11:39:35.998380+0000","last_clean":"2024-03-24T11:39:35.998380+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:39:35.998380+0000","last_undegraded":"2024-03-24T11:39:35.998380+0000","last_fullsized":"2024-03-24T11:39:35.998380+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:37:55.694628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.694628+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":436,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.383777+0000","last_change":"2024-03-24T11:39:28.579890+0000","last_active":"2024-03-24T11:39:39.383777+0000","last_peered":"2024-03-24T11:39:39.383777+0000","last_clean":"2024-03-24T11:39:39.383777+0000","last_became_active":"2024-03-24T11:39:28.575621+0000","last_became_peered":"2024-03-24T11:39:28.575621+0000","last_unstale":"2024-03-24T11:39:39.383777+0000","last_undegraded":"2024-03-24T11:39:39.383777+0000","last_fullsized":"2024-03-24T11:39:39.383777+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.172813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.172813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":796,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381376+0000","last_change":"2024-03-24T11:39:15.574465+0000","last_active":"2024-03-24T11:39:39.381376+0000","last_peered":"2024-03-24T11:39:39.381376+0000","last_clean":"2024-03-24T11:39:39.381376+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:39:39.381376+0000","last_undegraded":"2024-03-24T11:39:39.381376+0000","last_fullsized":"2024-03-24T11:39:39.381376+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:15.574327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:15.574327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:16.471082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":807,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.013936+0000","last_change":"2024-03-24T11:38:50.433428+0000","last_active":"2024-03-24T11:39:36.013936+0000","last_peered":"2024-03-24T11:39:36.013936+0000","last_clean":"2024-03-24T11:39:36.013936+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:39:36.013936+0000","last_undegraded":"2024-03-24T11:39:36.013936+0000","last_fullsized":"2024-03-24T11:39:36.013936+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:50.432934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:50.432934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:07.785937+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":853,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.390488+0000","last_change":"2024-03-24T11:39:28.598503+0000","last_active":"2024-03-24T11:39:39.390488+0000","last_peered":"2024-03-24T11:39:39.390488+0000","last_clean":"2024-03-24T11:39:39.390488+0000","last_became_active":"2024-03-24T11:39:28.575371+0000","last_became_peered":"2024-03-24T11:39:28.575371+0000","last_unstale":"2024-03-24T11:39:39.390488+0000","last_undegraded":"2024-03-24T11:39:39.390488+0000","last_fullsized":"2024-03-24T11:39:39.390488+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:33:40.545997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:33:40.545997+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":833,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008717+0000","last_change":"2024-03-24T11:39:28.574683+0000","last_active":"2024-03-24T11:39:36.008717+0000","last_peered":"2024-03-24T11:39:36.008717+0000","last_clean":"2024-03-24T11:39:36.008717+0000","last_became_active":"2024-03-24T11:39:28.574318+0000","last_became_peered":"2024-03-24T11:39:28.574318+0000","last_unstale":"2024-03-24T11:39:36.008717+0000","last_undegraded":"2024-03-24T11:39:36.008717+0000","last_fullsized":"2024-03-24T11:39:36.008717+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.913497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.913497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:01.674488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":129,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008847+0000","last_change":"2024-03-24T11:39:28.552174+0000","last_active":"2024-03-24T11:39:36.008847+0000","last_peered":"2024-03-24T11:39:36.008847+0000","last_clean":"2024-03-24T11:39:36.008847+0000","last_became_active":"2024-03-24T11:39:28.551550+0000","last_became_peered":"2024-03-24T11:39:28.551550+0000","last_unstale":"2024-03-24T11:39:36.008847+0000","last_undegraded":"2024-03-24T11:39:36.008847+0000","last_fullsized":"2024-03-24T11:39:36.008847+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_clean_scrub_stamp":"2024-03-24T11:39:06.116941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:23.859822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":792,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381719+0000","last_change":"2024-03-24T11:36:48.160280+0000","last_active":"2024-03-24T11:39:39.381719+0000","last_peered":"2024-03-24T11:39:39.381719+0000","last_clean":"2024-03-24T11:39:39.381719+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:39:39.381719+0000","last_undegraded":"2024-03-24T11:39:39.381719+0000","last_fullsized":"2024-03-24T11:39:39.381719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:48.160164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:48.160164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"421'14","reported_seq":820,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.382911+0000","last_change":"2024-03-24T11:38:54.224427+0000","last_active":"2024-03-24T11:39:39.382911+0000","last_peered":"2024-03-24T11:39:39.382911+0000","last_clean":"2024-03-24T11:39:39.382911+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:39:39.382911+0000","last_undegraded":"2024-03-24T11:39:39.382911+0000","last_fullsized":"2024-03-24T11:39:39.382911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:38:54.222100+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:38:54.222100+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:06.658198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":827,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.016464+0000","last_change":"2024-03-24T11:38:53.264039+0000","last_active":"2024-03-24T11:39:36.016464+0000","last_peered":"2024-03-24T11:39:36.016464+0000","last_clean":"2024-03-24T11:39:36.016464+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:39:36.016464+0000","last_undegraded":"2024-03-24T11:39:36.016464+0000","last_fullsized":"2024-03-24T11:39:36.016464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.263761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.263761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:18.212346+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":847,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009553+0000","last_change":"2024-03-24T11:39:28.572502+0000","last_active":"2024-03-24T11:39:36.009553+0000","last_peered":"2024-03-24T11:39:36.009553+0000","last_clean":"2024-03-24T11:39:36.009553+0000","last_became_active":"2024-03-24T11:39:28.572151+0000","last_became_peered":"2024-03-24T11:39:28.572151+0000","last_unstale":"2024-03-24T11:39:36.009553+0000","last_undegraded":"2024-03-24T11:39:36.009553+0000","last_fullsized":"2024-03-24T11:39:36.009553+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.997698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.997698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:03.279264+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":120,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.388169+0000","last_change":"2024-03-24T11:39:28.588074+0000","last_active":"2024-03-24T11:39:39.388169+0000","last_peered":"2024-03-24T11:39:39.388169+0000","last_clean":"2024-03-24T11:39:39.388169+0000","last_became_active":"2024-03-24T11:39:28.585224+0000","last_became_peered":"2024-03-24T11:39:28.585224+0000","last_unstale":"2024-03-24T11:39:39.388169+0000","last_undegraded":"2024-03-24T11:39:39.388169+0000","last_fullsized":"2024-03-24T11:39:39.388169+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:06.147813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:06.147813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"421'32","reported_seq":888,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.010708+0000","last_change":"2024-03-24T11:38:11.032460+0000","last_active":"2024-03-24T11:39:36.010708+0000","last_peered":"2024-03-24T11:39:36.010708+0000","last_clean":"2024-03-24T11:39:36.010708+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:39:36.010708+0000","last_undegraded":"2024-03-24T11:39:36.010708+0000","last_fullsized":"2024-03-24T11:39:36.010708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'24","last_scrub_stamp":"2024-03-24T11:38:11.032318+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.032318+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":804,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.382696+0000","last_change":"2024-03-24T11:38:01.211155+0000","last_active":"2024-03-24T11:39:39.382696+0000","last_peered":"2024-03-24T11:39:39.382696+0000","last_clean":"2024-03-24T11:39:39.382696+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:39:39.382696+0000","last_undegraded":"2024-03-24T11:39:39.382696+0000","last_fullsized":"2024-03-24T11:39:39.382696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.211052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.211052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":819,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.018497+0000","last_change":"2024-03-24T11:39:28.576830+0000","last_active":"2024-03-24T11:39:36.018497+0000","last_peered":"2024-03-24T11:39:36.018497+0000","last_clean":"2024-03-24T11:39:36.018497+0000","last_became_active":"2024-03-24T11:39:28.576207+0000","last_became_peered":"2024-03-24T11:39:28.576207+0000","last_unstale":"2024-03-24T11:39:36.018497+0000","last_undegraded":"2024-03-24T11:39:36.018497+0000","last_fullsized":"2024-03-24T11:39:36.018497+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:59.968275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:37:59.968275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":829,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.002943+0000","last_change":"2024-03-24T11:38:12.027553+0000","last_active":"2024-03-24T11:39:36.002943+0000","last_peered":"2024-03-24T11:39:36.002943+0000","last_clean":"2024-03-24T11:39:36.002943+0000","last_became_active":"2024-03-24T11:36:58.592355+0000","last_became_peered":"2024-03-24T11:36:58.592355+0000","last_unstale":"2024-03-24T11:39:36.002943+0000","last_undegraded":"2024-03-24T11:39:36.002943+0000","last_fullsized":"2024-03-24T11:39:36.002943+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.027430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'30","reported_seq":906,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.018274+0000","last_change":"2024-03-24T11:39:28.605333+0000","last_active":"2024-03-24T11:39:36.018274+0000","last_peered":"2024-03-24T11:39:36.018274+0000","last_clean":"2024-03-24T11:39:36.018274+0000","last_became_active":"2024-03-24T11:39:28.576661+0000","last_became_peered":"2024-03-24T11:39:28.576661+0000","last_unstale":"2024-03-24T11:39:36.018274+0000","last_undegraded":"2024-03-24T11:39:36.018274+0000","last_fullsized":"2024-03-24T11:39:36.018274+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:37:36.695829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:36.695829+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":846,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998750+0000","last_change":"2024-03-24T11:37:52.677447+0000","last_active":"2024-03-24T11:39:35.998750+0000","last_peered":"2024-03-24T11:39:35.998750+0000","last_clean":"2024-03-24T11:39:35.998750+0000","last_became_active":"2024-03-24T11:36:58.593112+0000","last_became_peered":"2024-03-24T11:36:58.593112+0000","last_unstale":"2024-03-24T11:39:35.998750+0000","last_undegraded":"2024-03-24T11:39:35.998750+0000","last_fullsized":"2024-03-24T11:39:35.998750+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:37:52.677294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.677294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"429'11","reported_seq":1120,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.010555+0000","last_change":"2024-03-24T11:39:28.599069+0000","last_active":"2024-03-24T11:39:36.010555+0000","last_peered":"2024-03-24T11:39:36.010555+0000","last_clean":"2024-03-24T11:39:36.010555+0000","last_became_active":"2024-03-24T11:39:28.579771+0000","last_became_peered":"2024-03-24T11:39:28.579771+0000","last_unstale":"2024-03-24T11:39:36.010555+0000","last_undegraded":"2024-03-24T11:39:36.010555+0000","last_fullsized":"2024-03-24T11:39:36.010555+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'10","last_scrub_stamp":"2024-03-24T11:38:04.870822+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.870822+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":488,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008096+0000","last_change":"2024-03-24T11:39:28.582106+0000","last_active":"2024-03-24T11:39:36.008096+0000","last_peered":"2024-03-24T11:39:36.008096+0000","last_clean":"2024-03-24T11:39:36.008096+0000","last_became_active":"2024-03-24T11:39:28.562081+0000","last_became_peered":"2024-03-24T11:39:28.562081+0000","last_unstale":"2024-03-24T11:39:36.008096+0000","last_undegraded":"2024-03-24T11:39:36.008096+0000","last_fullsized":"2024-03-24T11:39:36.008096+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:39:19.096734+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:39:19.096734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:42.009845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"421'24","reported_seq":851,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.383984+0000","last_change":"2024-03-24T11:36:59.961408+0000","last_active":"2024-03-24T11:39:39.383984+0000","last_peered":"2024-03-24T11:39:39.383984+0000","last_clean":"2024-03-24T11:39:39.383984+0000","last_became_active":"2024-03-24T11:36:58.591714+0000","last_became_peered":"2024-03-24T11:36:58.591714+0000","last_unstale":"2024-03-24T11:39:39.383984+0000","last_undegraded":"2024-03-24T11:39:39.383984+0000","last_fullsized":"2024-03-24T11:39:39.383984+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:36:59.961306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:59.961306+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":113,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998076+0000","last_change":"2024-03-24T11:37:50.620288+0000","last_active":"2024-03-24T11:39:35.998076+0000","last_peered":"2024-03-24T11:39:35.998076+0000","last_clean":"2024-03-24T11:39:35.998076+0000","last_became_active":"2024-03-24T11:36:58.597448+0000","last_became_peered":"2024-03-24T11:36:58.597448+0000","last_unstale":"2024-03-24T11:39:35.998076+0000","last_undegraded":"2024-03-24T11:39:35.998076+0000","last_fullsized":"2024-03-24T11:39:35.998076+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:50.620171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:50.620171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":122,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381757+0000","last_change":"2024-03-24T11:39:28.581090+0000","last_active":"2024-03-24T11:39:39.381757+0000","last_peered":"2024-03-24T11:39:39.381757+0000","last_clean":"2024-03-24T11:39:39.381757+0000","last_became_active":"2024-03-24T11:39:28.577496+0000","last_became_peered":"2024-03-24T11:39:28.577496+0000","last_unstale":"2024-03-24T11:39:39.381757+0000","last_undegraded":"2024-03-24T11:39:39.381757+0000","last_fullsized":"2024-03-24T11:39:39.381757+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.066063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.066063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":788,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.380739+0000","last_change":"2024-03-24T11:35:49.842222+0000","last_active":"2024-03-24T11:39:39.380739+0000","last_peered":"2024-03-24T11:39:39.380739+0000","last_clean":"2024-03-24T11:39:39.380739+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:39:39.380739+0000","last_undegraded":"2024-03-24T11:39:39.380739+0000","last_fullsized":"2024-03-24T11:39:39.380739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:49.842117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:35:49.842117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"421'39","reported_seq":908,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009466+0000","last_change":"2024-03-24T11:39:15.286788+0000","last_active":"2024-03-24T11:39:36.009466+0000","last_peered":"2024-03-24T11:39:36.009466+0000","last_clean":"2024-03-24T11:39:36.009466+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:39:36.009466+0000","last_undegraded":"2024-03-24T11:39:36.009466+0000","last_fullsized":"2024-03-24T11:39:36.009466+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:39:15.286646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:15.286646+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:24.368123+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":839,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.016953+0000","last_change":"2024-03-24T11:39:28.573154+0000","last_active":"2024-03-24T11:39:36.016953+0000","last_peered":"2024-03-24T11:39:36.016953+0000","last_clean":"2024-03-24T11:39:36.016953+0000","last_became_active":"2024-03-24T11:39:28.572672+0000","last_became_peered":"2024-03-24T11:39:28.572672+0000","last_unstale":"2024-03-24T11:39:36.016953+0000","last_undegraded":"2024-03-24T11:39:36.016953+0000","last_fullsized":"2024-03-24T11:39:36.016953+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:05.380874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:39:05.380874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:18.812059+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":825,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.002780+0000","last_change":"2024-03-24T11:37:43.684412+0000","last_active":"2024-03-24T11:39:36.002780+0000","last_peered":"2024-03-24T11:39:36.002780+0000","last_clean":"2024-03-24T11:39:36.002780+0000","last_became_active":"2024-03-24T11:36:58.596592+0000","last_became_peered":"2024-03-24T11:36:58.596592+0000","last_unstale":"2024-03-24T11:39:36.002780+0000","last_undegraded":"2024-03-24T11:39:36.002780+0000","last_fullsized":"2024-03-24T11:39:36.002780+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.684286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.684286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":113,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.011095+0000","last_change":"2024-03-24T11:37:48.869860+0000","last_active":"2024-03-24T11:39:36.011095+0000","last_peered":"2024-03-24T11:39:36.011095+0000","last_clean":"2024-03-24T11:39:36.011095+0000","last_became_active":"2024-03-24T11:36:58.575356+0000","last_became_peered":"2024-03-24T11:36:58.575356+0000","last_unstale":"2024-03-24T11:39:36.011095+0000","last_undegraded":"2024-03-24T11:39:36.011095+0000","last_fullsized":"2024-03-24T11:39:36.011095+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:48.869684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:37:48.869684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"421'9","reported_seq":472,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381181+0000","last_change":"2024-03-24T11:36:58.602264+0000","last_active":"2024-03-24T11:39:39.381181+0000","last_peered":"2024-03-24T11:39:39.381181+0000","last_clean":"2024-03-24T11:39:39.381181+0000","last_became_active":"2024-03-24T11:36:58.586221+0000","last_became_peered":"2024-03-24T11:36:58.586221+0000","last_unstale":"2024-03-24T11:39:39.381181+0000","last_undegraded":"2024-03-24T11:39:39.381181+0000","last_fullsized":"2024-03-24T11:39:39.381181+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:36:04.265719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:04.265719+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":825,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008246+0000","last_change":"2024-03-24T11:37:41.153332+0000","last_active":"2024-03-24T11:39:36.008246+0000","last_peered":"2024-03-24T11:39:36.008246+0000","last_clean":"2024-03-24T11:39:36.008246+0000","last_became_active":"2024-03-24T11:36:58.585301+0000","last_became_peered":"2024-03-24T11:36:58.585301+0000","last_unstale":"2024-03-24T11:39:36.008246+0000","last_undegraded":"2024-03-24T11:39:36.008246+0000","last_fullsized":"2024-03-24T11:39:36.008246+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.153221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.153221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":121,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.997924+0000","last_change":"2024-03-24T11:39:28.572349+0000","last_active":"2024-03-24T11:39:35.997924+0000","last_peered":"2024-03-24T11:39:35.997924+0000","last_clean":"2024-03-24T11:39:35.997924+0000","last_became_active":"2024-03-24T11:39:28.571768+0000","last_became_peered":"2024-03-24T11:39:28.571768+0000","last_unstale":"2024-03-24T11:39:35.997924+0000","last_undegraded":"2024-03-24T11:39:35.997924+0000","last_fullsized":"2024-03-24T11:39:35.997924+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.674916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.674916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":807,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008008+0000","last_change":"2024-03-24T11:37:46.149252+0000","last_active":"2024-03-24T11:39:36.008008+0000","last_peered":"2024-03-24T11:39:36.008008+0000","last_clean":"2024-03-24T11:39:36.008008+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:39:36.008008+0000","last_undegraded":"2024-03-24T11:39:36.008008+0000","last_fullsized":"2024-03-24T11:39:36.008008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:46.149176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:37:46.149176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"421'21","reported_seq":494,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.389145+0000","last_change":"2024-03-24T11:38:53.240408+0000","last_active":"2024-03-24T11:39:39.389145+0000","last_peered":"2024-03-24T11:39:39.389145+0000","last_clean":"2024-03-24T11:39:39.389145+0000","last_became_active":"2024-03-24T11:36:58.580893+0000","last_became_peered":"2024-03-24T11:36:58.580893+0000","last_unstale":"2024-03-24T11:39:39.389145+0000","last_undegraded":"2024-03-24T11:39:39.389145+0000","last_fullsized":"2024-03-24T11:39:39.389145+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:53.238808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.238808+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:12.777399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":833,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009105+0000","last_change":"2024-03-24T11:39:07.004625+0000","last_active":"2024-03-24T11:39:36.009105+0000","last_peered":"2024-03-24T11:39:36.009105+0000","last_clean":"2024-03-24T11:39:36.009105+0000","last_became_active":"2024-03-24T11:36:58.590154+0000","last_became_peered":"2024-03-24T11:36:58.590154+0000","last_unstale":"2024-03-24T11:39:36.009105+0000","last_undegraded":"2024-03-24T11:39:36.009105+0000","last_fullsized":"2024-03-24T11:39:36.009105+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:07.004520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:39:07.004520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:16.355297+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":831,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.008498+0000","last_change":"2024-03-24T11:39:28.562677+0000","last_active":"2024-03-24T11:39:36.008498+0000","last_peered":"2024-03-24T11:39:36.008498+0000","last_clean":"2024-03-24T11:39:36.008498+0000","last_became_active":"2024-03-24T11:39:28.562146+0000","last_became_peered":"2024-03-24T11:39:28.562146+0000","last_unstale":"2024-03-24T11:39:36.008498+0000","last_undegraded":"2024-03-24T11:39:36.008498+0000","last_fullsized":"2024-03-24T11:39:36.008498+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:54.088400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:37:54.088400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":813,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.014264+0000","last_change":"2024-03-24T11:39:28.549287+0000","last_active":"2024-03-24T11:39:36.014264+0000","last_peered":"2024-03-24T11:39:36.014264+0000","last_clean":"2024-03-24T11:39:36.014264+0000","last_became_active":"2024-03-24T11:39:28.548958+0000","last_became_peered":"2024-03-24T11:39:28.548958+0000","last_unstale":"2024-03-24T11:39:36.014264+0000","last_undegraded":"2024-03-24T11:39:36.014264+0000","last_fullsized":"2024-03-24T11:39:36.014264+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:31.701661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:31.701661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'39","reported_seq":853,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.390665+0000","last_change":"2024-03-24T11:37:02.002116+0000","last_active":"2024-03-24T11:39:39.390665+0000","last_peered":"2024-03-24T11:39:39.390665+0000","last_clean":"2024-03-24T11:39:39.390665+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:39:39.390665+0000","last_undegraded":"2024-03-24T11:39:39.390665+0000","last_fullsized":"2024-03-24T11:39:39.390665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:37:02.002019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:02.002019+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":798,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.386037+0000","last_change":"2024-03-24T11:35:51.761976+0000","last_active":"2024-03-24T11:39:39.386037+0000","last_peered":"2024-03-24T11:39:39.386037+0000","last_clean":"2024-03-24T11:39:39.386037+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:39:39.386037+0000","last_undegraded":"2024-03-24T11:39:39.386037+0000","last_fullsized":"2024-03-24T11:39:39.386037+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:51.761850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:51.761850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":122,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009026+0000","last_change":"2024-03-24T11:39:11.310606+0000","last_active":"2024-03-24T11:39:36.009026+0000","last_peered":"2024-03-24T11:39:36.009026+0000","last_clean":"2024-03-24T11:39:36.009026+0000","last_became_active":"2024-03-24T11:36:58.584540+0000","last_became_peered":"2024-03-24T11:36:58.584540+0000","last_unstale":"2024-03-24T11:39:36.009026+0000","last_undegraded":"2024-03-24T11:39:36.009026+0000","last_fullsized":"2024-03-24T11:39:36.009026+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:11.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:11.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:22.016627+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":790,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-24T11:39:39.381470+0000","last_change":"2024-03-24T11:39:28.584057+0000","last_active":"2024-03-24T11:39:39.381470+0000","last_peered":"2024-03-24T11:39:39.381470+0000","last_clean":"2024-03-24T11:39:39.381470+0000","last_became_active":"2024-03-24T11:39:28.580757+0000","last_became_peered":"2024-03-24T11:39:28.580757+0000","last_unstale":"2024-03-24T11:39:39.381470+0000","last_undegraded":"2024-03-24T11:39:39.381470+0000","last_fullsized":"2024-03-24T11:39:39.381470+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:53.164514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:53.164514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":524,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.009479+0000","last_change":"2024-03-24T11:39:17.302385+0000","last_active":"2024-03-24T11:39:36.009479+0000","last_peered":"2024-03-24T11:39:36.009479+0000","last_clean":"2024-03-24T11:39:36.009479+0000","last_became_active":"2024-03-24T11:36:58.580796+0000","last_became_peered":"2024-03-24T11:36:58.580796+0000","last_unstale":"2024-03-24T11:39:36.009479+0000","last_undegraded":"2024-03-24T11:39:36.009479+0000","last_fullsized":"2024-03-24T11:39:36.009479+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:39:17.302257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:17.302257+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:29.172499+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":820,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.016100+0000","last_change":"2024-03-24T11:38:08.039863+0000","last_active":"2024-03-24T11:39:36.016100+0000","last_peered":"2024-03-24T11:39:36.016100+0000","last_clean":"2024-03-24T11:39:36.016100+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:39:36.016100+0000","last_undegraded":"2024-03-24T11:39:36.016100+0000","last_fullsized":"2024-03-24T11:39:36.016100+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:08.039401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.039401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:38.295907+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":825,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:35.998842+0000","last_change":"2024-03-24T11:39:28.558372+0000","last_active":"2024-03-24T11:39:35.998842+0000","last_peered":"2024-03-24T11:39:35.998842+0000","last_clean":"2024-03-24T11:39:35.998842+0000","last_became_active":"2024-03-24T11:39:28.557876+0000","last_became_peered":"2024-03-24T11:39:28.557876+0000","last_unstale":"2024-03-24T11:39:35.998842+0000","last_undegraded":"2024-03-24T11:39:35.998842+0000","last_fullsized":"2024-03-24T11:39:35.998842+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.752016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.752016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":119,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-24T11:39:36.007929+0000","last_change":"2024-03-24T11:39:28.546953+0000","last_active":"2024-03-24T11:39:36.007929+0000","last_peered":"2024-03-24T11:39:36.007929+0000","last_clean":"2024-03-24T11:39:36.007929+0000","last_became_active":"2024-03-24T11:39:28.546592+0000","last_became_peered":"2024-03-24T11:39:28.546592+0000","last_unstale":"2024-03-24T11:39:36.007929+0000","last_undegraded":"2024-03-24T11:39:36.007929+0000","last_fullsized":"2024-03-24T11:39:36.007929+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.062184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.062184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3322871832,"num_objects":50704,"num_object_clones":0,"num_object_copies":152112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14469,"num_objects_unfound":0,"num_objects_dirty":50704,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189888,"num_write_kb":7698881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8539,"num_bytes_recovered":559611904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10577653760,"data_stored":10577641544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59566,"ondisk_log_size":59566,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":543,"num_read_kb":538,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":5568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":3856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":13944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1026,"ondisk_log_size":1026,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365058,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2398196,"kb_used_data":1721452,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":91346956,"statfs":{"total":95995035648,"available":93539282944,"internally_reserved":0,"allocated":1762766848,"data_stored":1762226227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":692976070},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575132,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2801120,"kb_used_data":1907796,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":90944032,"statfs":{"total":95995035648,"available":93126688768,"internally_reserved":0,"allocated":1953583104,"data_stored":1953002296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738657,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2251440,"kb_used_data":1439976,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":91493712,"statfs":{"total":95995035648,"available":93689561088,"internally_reserved":0,"allocated":1474535424,"data_stored":1473997967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411607,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1927988,"kb_used_data":1233192,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":91817164,"statfs":{"total":95995035648,"available":94020775936,"internally_reserved":0,"allocated":1262788608,"data_stored":1262251546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149968,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2394972,"kb_used_data":1555664,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":91350180,"statfs":{"total":95995035648,"available":93542584320,"internally_reserved":0,"allocated":1592999936,"data_stored":1592432535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":859306439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659399,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1880972,"kb_used_data":1065408,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91864180,"statfs":{"total":95995035648,"available":94068920320,"internally_reserved":0,"allocated":1090977792,"data_stored":1090417002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1461846016,"data_stored":1461846016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1922830336,"data_stored":1922826264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1248923648,"data_stored":1248919576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1631518720,"data_stored":1631518720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1092222976,"data_stored":1092222976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1753026560,"data_stored":1753022488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1467285504,"data_stored":1467285504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:39:42.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-03-24T11:39:42.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:42.644 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:42.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:42.651 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:42 smithi181 ceph-mon[33998]: from='client.25817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:39:42.694 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:42 smithi179 ceph-mon[29608]: from='client.25817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:39:42.695 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:42 smithi179 ceph-mon[37907]: from='client.25817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:39:42.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:42.775 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:42.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:42.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:42.906 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:42.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:43.008 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:43.053 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:43.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:43.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:43.220 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:43.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:43.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:43.364 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:43.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:43.466 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:43.503 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:43.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:43.604 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:43.666 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:43.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:43 smithi179 ceph-mon[29608]: pgmap v961: 158 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 151 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 746 op/s; 14469/152706 objects misplaced (9.475%); 43 MiB/s, 681 objects/s recovering 2024-03-24T11:39:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:43 smithi179 ceph-mon[37907]: pgmap v961: 158 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 151 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 746 op/s; 14469/152706 objects misplaced (9.475%); 43 MiB/s, 681 objects/s recovering 2024-03-24T11:39:43.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:43.811 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:43.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:43 smithi181 ceph-mon[33998]: pgmap v961: 158 pgs: 1 active+remapped+backfill_wait, 6 active+remapped+backfilling, 151 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 790 B/s rd, 0 B/s wr, 746 op/s; 14469/152706 objects misplaced (9.475%); 43 MiB/s, 681 objects/s recovering 2024-03-24T11:39:43.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:43.943 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:43.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:44.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:44.068 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:44.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:44.170 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:44.204 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:44.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:44.305 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:44.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:44.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:44.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:44.482 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:44.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:44.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:44.610 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:44.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:44.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:39:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:39:44.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:44.744 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:44.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:44.846 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:44.880 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:44.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:44.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:45.013 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:45.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:45.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:45.146 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:45.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:45.248 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:45.292 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:45.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:45.374 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:39:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:39:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:39:45.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:45.438 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:45.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:45.540 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:45.582 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:45.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:45 smithi181 ceph-mon[33998]: osdmap e437: 8 total, 8 up, 6 in 2024-03-24T11:39:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:45 smithi181 ceph-mon[33998]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.28k op/s; 9207/144189 objects misplaced (6.385%); 91 MiB/s, 1.46k objects/s recovering 2024-03-24T11:39:45.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:45.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:45 smithi179 ceph-mon[29608]: osdmap e437: 8 total, 8 up, 6 in 2024-03-24T11:39:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:45 smithi179 ceph-mon[29608]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.28k op/s; 9207/144189 objects misplaced (6.385%); 91 MiB/s, 1.46k objects/s recovering 2024-03-24T11:39:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:45 smithi179 ceph-mon[37907]: osdmap e437: 8 total, 8 up, 6 in 2024-03-24T11:39:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:45 smithi179 ceph-mon[37907]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.28k op/s; 9207/144189 objects misplaced (6.385%); 91 MiB/s, 1.46k objects/s recovering 2024-03-24T11:39:45.728 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:45.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:45.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:45.865 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:45.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:45.966 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:46.005 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:46.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:46.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:46.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:46.282 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:46.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.408 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:46.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:46.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.551 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:46.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:46.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:46.681 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:46.783 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.824 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:46.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:46.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:46.952 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:46.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:47.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:47.085 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:47.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:47.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:47.222 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:47.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:47.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:47.355 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:47.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:47.457 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:47.496 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:47.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:47.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:47.622 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:47.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:47.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:47.756 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:47.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:47.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:47.884 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:47.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:47.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:47 smithi181 ceph-mon[33998]: pgmap v964: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 719 B/s rd, 0 B/s wr, 797 op/s; 9207/139764 objects misplaced (6.588%); 38 MiB/s, 607 objects/s recovering 2024-03-24T11:39:47.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:47 smithi179 ceph-mon[29608]: pgmap v964: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 719 B/s rd, 0 B/s wr, 797 op/s; 9207/139764 objects misplaced (6.588%); 38 MiB/s, 607 objects/s recovering 2024-03-24T11:39:47.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:47 smithi179 ceph-mon[37907]: pgmap v964: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 719 B/s rd, 0 B/s wr, 797 op/s; 9207/139764 objects misplaced (6.588%); 38 MiB/s, 607 objects/s recovering 2024-03-24T11:39:47.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:48.016 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:48.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:48.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:48.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.280 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:48.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:48.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:48.427 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:48.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:48.559 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:48.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:48.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:48.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:48.832 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:48.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:48.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:48 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:39:48.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:48 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:39:48.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:48 smithi179 ceph-mon[29608]: osdmap e438: 8 total, 8 up, 6 in 2024-03-24T11:39:48.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:48 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:39:48.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:48 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:39:48.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:48 smithi179 ceph-mon[37907]: osdmap e438: 8 total, 8 up, 6 in 2024-03-24T11:39:48.963 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:48.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:49.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:49.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:49.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:49.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:48 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:39:49.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:48 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:39:49.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:48 smithi181 ceph-mon[33998]: osdmap e438: 8 total, 8 up, 6 in 2024-03-24T11:39:49.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:49.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:49.229 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:49.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:49.329 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:49.354 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:49.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:49.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:49.483 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:49.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:49.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:49.558 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:49.558 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:39:49.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:49.659 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:49.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:49.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:49.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:49.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:49.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:49.924 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:49.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:49.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[29608]: pgmap v966: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s; 9207/139764 objects misplaced (6.588%); 34 MiB/s, 540 objects/s recovering 2024-03-24T11:39:49.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:49.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:39:49.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:39:49.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[29608]: osdmap e439: 8 total, 8 up, 6 in 2024-03-24T11:39:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[37907]: pgmap v966: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s; 9207/139764 objects misplaced (6.588%); 34 MiB/s, 540 objects/s recovering 2024-03-24T11:39:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:39:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:39:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:49 smithi179 ceph-mon[37907]: osdmap e439: 8 total, 8 up, 6 in 2024-03-24T11:39:50.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:50.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.057 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:50.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:50.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:49 smithi181 ceph-mon[33998]: pgmap v966: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 539 op/s; 9207/139764 objects misplaced (6.588%); 34 MiB/s, 540 objects/s recovering 2024-03-24T11:39:50.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:39:50.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:49 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:39:50.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:49 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:39:50.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:49 smithi181 ceph-mon[33998]: osdmap e439: 8 total, 8 up, 6 in 2024-03-24T11:39:50.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:50.188 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:50.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:50.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.331 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:50.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:50.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:50.463 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:50.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:50.603 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:50.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:50.733 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:50.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:50 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:39:50.810 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:50 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:39:50.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:50 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:39:50.810 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:50 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:39:50.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:50.861 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:50.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:50.991 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:50.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:51.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:51.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:51.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:51.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:50 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:39:51.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:50 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:39:51.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:51.244 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:51.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:51.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:51.372 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:51.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:51.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:51.498 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:51.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:51.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:51.625 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:51.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:51.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:51.768 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:51.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:51.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:51.904 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:51.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:51.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:51 smithi179 ceph-mon[29608]: pgmap v968: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 837 B/s rd, 0 B/s wr, 982 op/s; 5860/126183 objects misplaced (4.644%); 26 MiB/s, 419 objects/s recovering 2024-03-24T11:39:51.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:51 smithi179 ceph-mon[29608]: osdmap e440: 8 total, 8 up, 6 in 2024-03-24T11:39:51.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:51 smithi179 ceph-mon[37907]: pgmap v968: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 837 B/s rd, 0 B/s wr, 982 op/s; 5860/126183 objects misplaced (4.644%); 26 MiB/s, 419 objects/s recovering 2024-03-24T11:39:51.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:51 smithi179 ceph-mon[37907]: osdmap e440: 8 total, 8 up, 6 in 2024-03-24T11:39:52.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:52.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:52.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:51 smithi181 ceph-mon[33998]: pgmap v968: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 837 B/s rd, 0 B/s wr, 982 op/s; 5860/126183 objects misplaced (4.644%); 26 MiB/s, 419 objects/s recovering 2024-03-24T11:39:52.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:51 smithi181 ceph-mon[33998]: osdmap e440: 8 total, 8 up, 6 in 2024-03-24T11:39:52.158 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:52.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:52.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:52.385 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:52.417 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:52.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:52.543 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:52.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:52.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.669 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:52.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:52.770 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.802 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:52.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:52.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:52.911 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:39:52.912 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:39:52.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:52 smithi179 ceph-mon[37907]: osdmap e441: 8 total, 8 up, 6 in 2024-03-24T11:39:52.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:52 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2349618511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:52.913 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:52 smithi179 ceph-mon[37907]: osdmap e442: 8 total, 8 up, 6 in 2024-03-24T11:39:52.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:52 smithi179 ceph-mon[29608]: osdmap e441: 8 total, 8 up, 6 in 2024-03-24T11:39:52.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:52 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2349618511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:52.914 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:52 smithi179 ceph-mon[29608]: osdmap e442: 8 total, 8 up, 6 in 2024-03-24T11:39:52.928 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:52.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:53.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:53.054 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:52 smithi181 ceph-mon[33998]: osdmap e441: 8 total, 8 up, 6 in 2024-03-24T11:39:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:52 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2349618511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:39:53.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:52 smithi181 ceph-mon[33998]: osdmap e442: 8 total, 8 up, 6 in 2024-03-24T11:39:53.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:53.185 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:53.286 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.313 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:53.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:53.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:53.439 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:53.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.566 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:53.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:53.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:53.692 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:53.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:53.818 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:53.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:53.945 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:53.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:53 smithi179 ceph-mon[29608]: pgmap v971: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 940 op/s; 5860/126183 objects misplaced (4.644%); 33 MiB/s, 532 objects/s recovering 2024-03-24T11:39:53.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:53 smithi179 ceph-mon[29608]: osdmap e443: 8 total, 8 up, 6 in 2024-03-24T11:39:53.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:53 smithi179 ceph-mon[37907]: pgmap v971: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 940 op/s; 5860/126183 objects misplaced (4.644%); 33 MiB/s, 532 objects/s recovering 2024-03-24T11:39:53.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:53 smithi179 ceph-mon[37907]: osdmap e443: 8 total, 8 up, 6 in 2024-03-24T11:39:54.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.070 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:54.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:53 smithi181 ceph-mon[33998]: pgmap v971: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 152 active+clean; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 940 op/s; 5860/126183 objects misplaced (4.644%); 33 MiB/s, 532 objects/s recovering 2024-03-24T11:39:54.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:53 smithi181 ceph-mon[33998]: osdmap e443: 8 total, 8 up, 6 in 2024-03-24T11:39:54.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:54.195 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:54.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.321 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:54.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:54.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.448 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:54.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:54.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.573 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:54.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:54.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:39:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:39:54.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:54.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:54.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:54.841 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:54.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:54.985 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:54.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:55.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:55.126 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:55.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:55.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:39:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:39:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:39:55.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:55.262 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:55.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:55.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:55.403 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:55.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:55.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:55.534 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:55.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:55.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:55.661 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:55.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:55.763 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:55.789 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:55.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:55.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:55.918 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:55.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:55.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:55.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:55 smithi179 ceph-mon[29608]: pgmap v974: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 484 op/s; 1460/120381 objects misplaced (1.213%); 63 MiB/s, 1.01k objects/s recovering 2024-03-24T11:39:55.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:55 smithi179 ceph-mon[29608]: osdmap e444: 8 total, 8 up, 6 in 2024-03-24T11:39:55.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:55.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:55 smithi179 ceph-mon[37907]: pgmap v974: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 484 op/s; 1460/120381 objects misplaced (1.213%); 63 MiB/s, 1.01k objects/s recovering 2024-03-24T11:39:55.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:55 smithi179 ceph-mon[37907]: osdmap e444: 8 total, 8 up, 6 in 2024-03-24T11:39:56.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:56.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:56.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:56.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:56.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:39:56.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:55 smithi181 ceph-mon[33998]: pgmap v974: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 484 op/s; 1460/120381 objects misplaced (1.213%); 63 MiB/s, 1.01k objects/s recovering 2024-03-24T11:39:56.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:55 smithi181 ceph-mon[33998]: osdmap e444: 8 total, 8 up, 6 in 2024-03-24T11:39:56.186 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:56.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:56.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:56.312 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:56.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:56.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:56.452 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:56.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:56.554 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:56.595 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:56.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:56.696 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:56.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:56.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:56.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:56.867 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:56.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:56.969 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:57.013 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:57.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:39:57.143 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:39:57.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:57.273 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:57.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:57.416 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:57.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.559 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:57.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:57.660 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:57.701 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:57.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.852 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:57.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:57.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:57 smithi179 ceph-mon[37907]: pgmap v976: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 529 op/s; 1460/118584 objects misplaced (1.231%); 53 MiB/s, 847 objects/s recovering 2024-03-24T11:39:57.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:57 smithi179 ceph-mon[29608]: pgmap v976: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 529 op/s; 1460/118584 objects misplaced (1.231%); 53 MiB/s, 847 objects/s recovering 2024-03-24T11:39:57.993 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:57.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:58.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:58.128 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:58.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:39:58.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:57 smithi181 ceph-mon[33998]: pgmap v976: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 529 op/s; 1460/118584 objects misplaced (1.231%); 53 MiB/s, 847 objects/s recovering 2024-03-24T11:39:58.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:58.256 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:58.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:58.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:58.381 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:58.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:58.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:58.510 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:58.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:58.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:58.638 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:58.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:58.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:39:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:58.783 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:58.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:39:58.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:58.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:58.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:59.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:59.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:59.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:39:59.168 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:39:59.270 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:39:59.312 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:39:59.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:39:59.456 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:39:59.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:59.586 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:59.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:39:59.715 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:39:59.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:39:59.845 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:39:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:39:59.947 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:39:59.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:39:59 smithi179 ceph-mon[29608]: pgmap v977: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 422 op/s; 1460/118584 objects misplaced (1.231%); 42 MiB/s, 676 objects/s recovering 2024-03-24T11:39:59.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:39:59 smithi179 ceph-mon[37907]: pgmap v977: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 422 op/s; 1460/118584 objects misplaced (1.231%); 42 MiB/s, 676 objects/s recovering 2024-03-24T11:39:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:39:59.991 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:39:59.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:00.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:00.099 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:40:00.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:00.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:00.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:39:59 smithi181 ceph-mon[33998]: pgmap v977: 158 pgs: 1 peering, 2 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 422 op/s; 1460/118584 objects misplaced (1.231%); 42 MiB/s, 676 objects/s recovering 2024-03-24T11:40:00.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:00.247 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:00.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:00.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:00.376 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:00.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:00.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:00.522 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:00.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:00.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:00.653 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:00.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:00.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:00.781 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:00.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:00.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:00.911 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:00.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:01.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:01.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:01.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:01.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:00 smithi181 ceph-mon[33998]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T11:40:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.169 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:01.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:01.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:00 smithi179 ceph-mon[29608]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T11:40:01.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:00 smithi179 ceph-mon[37907]: overall HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-24T11:40:01.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:01.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:01.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:01.423 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:01.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:01.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.552 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:01.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:01.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.678 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:01.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:01.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:01.803 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:01.904 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:01.946 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:01.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:01.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:01 smithi179 ceph-mon[29608]: pgmap v978: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 955 op/s; 583/103887 objects misplaced (0.561%); 39 MiB/s, 621 objects/s recovering 2024-03-24T11:40:01.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:01 smithi179 ceph-mon[37907]: pgmap v978: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 955 op/s; 583/103887 objects misplaced (0.561%); 39 MiB/s, 621 objects/s recovering 2024-03-24T11:40:02.049 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.078 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:02.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:02.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:01 smithi181 ceph-mon[33998]: pgmap v978: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 955 op/s; 583/103887 objects misplaced (0.561%); 39 MiB/s, 621 objects/s recovering 2024-03-24T11:40:02.179 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.223 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:02.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:02.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:02.352 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:02.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:02.480 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:02.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:02.627 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:02.729 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.755 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:02.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:02.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:02.896 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:02.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:02.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:03.024 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:03.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:03.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:03.152 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:03.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:03.254 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:03.294 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:03.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:03.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:03.439 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:03.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:03.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:03.568 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:03.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:03.670 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:03.713 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:03.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:03.814 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:03.864 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:03.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:03.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:03 smithi179 ceph-mon[29608]: pgmap v979: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 582 B/s rd, 0 B/s wr, 684 op/s; 583/103887 objects misplaced (0.561%); 7.1 MiB/s, 113 objects/s recovering 2024-03-24T11:40:03.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:03 smithi179 ceph-mon[37907]: pgmap v979: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 582 B/s rd, 0 B/s wr, 684 op/s; 583/103887 objects misplaced (0.561%); 7.1 MiB/s, 113 objects/s recovering 2024-03-24T11:40:03.965 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:04.004 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:04.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.134 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:04.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:04.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:03 smithi181 ceph-mon[33998]: pgmap v979: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 582 B/s rd, 0 B/s wr, 684 op/s; 583/103887 objects misplaced (0.561%); 7.1 MiB/s, 113 objects/s recovering 2024-03-24T11:40:04.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:04.292 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:04.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:04.433 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:04.479 INFO:teuthology.orchestra.run.smithi179.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:04.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:04.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:04.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:40:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:40:04.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:04.697 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:04.800 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:04.827 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:04.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:04.956 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:04.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:05.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:05.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:05.058 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:05.099 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:05.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:05.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:05.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:05.237 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:05.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:05.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:05.392 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:05.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:05.406 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:40:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:40:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:05.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:05.521 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:05.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:05.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:05.666 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:05.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:05.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:05.794 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:05.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:05.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:05.955 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:05.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:06.012 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:06.012 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:05 smithi179 ceph-mon[37907]: pgmap v980: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 934 op/s; 133/92373 objects misplaced (0.144%); 7.3 MiB/s, 117 objects/s recovering 2024-03-24T11:40:06.012 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:06.012 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:05 smithi179 ceph-mon[29608]: pgmap v980: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 934 op/s; 133/92373 objects misplaced (0.144%); 7.3 MiB/s, 117 objects/s recovering 2024-03-24T11:40:06.056 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:06.080 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:06.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:05 smithi181 ceph-mon[33998]: pgmap v980: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 934 op/s; 133/92373 objects misplaced (0.144%); 7.3 MiB/s, 117 objects/s recovering 2024-03-24T11:40:06.153 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:06.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:06.206 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:06.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:06.332 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:06.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.459 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:06.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:06.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.584 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:06.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:06.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T11:40:06.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.710 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:06.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:06.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:06.842 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:06.898 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:06.901 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:06 smithi181 ceph-mon[33998]: osdmap e445: 8 total, 8 up, 6 in 2024-03-24T11:40:06.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:06 smithi179 ceph-mon[29608]: osdmap e445: 8 total, 8 up, 6 in 2024-03-24T11:40:06.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:06 smithi179 ceph-mon[37907]: osdmap e445: 8 total, 8 up, 6 in 2024-03-24T11:40:06.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:06.973 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:06.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:07.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:07.106 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:07.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:07.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:07.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:07.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:07.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:07.385 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:07.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:07.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:07.518 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:07.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:07.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:07.644 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:07.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:07.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:07.770 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:07.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:07.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:07.917 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:07.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:08.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:08.058 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:08.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:08.076 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:07 smithi179 ceph-mon[29608]: pgmap v982: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 133/87105 objects misplaced (0.153%); 7.3 MiB/s, 117 objects/s recovering 2024-03-24T11:40:08.076 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:07 smithi179 ceph-mon[29608]: osdmap e446: 8 total, 8 up, 6 in 2024-03-24T11:40:08.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:07 smithi179 ceph-mon[37907]: pgmap v982: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 133/87105 objects misplaced (0.153%); 7.3 MiB/s, 117 objects/s recovering 2024-03-24T11:40:08.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:07 smithi179 ceph-mon[37907]: osdmap e446: 8 total, 8 up, 6 in 2024-03-24T11:40:08.082 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:07 smithi181 ceph-mon[33998]: pgmap v982: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 133/87105 objects misplaced (0.153%); 7.3 MiB/s, 117 objects/s recovering 2024-03-24T11:40:08.082 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:07 smithi181 ceph-mon[33998]: osdmap e446: 8 total, 8 up, 6 in 2024-03-24T11:40:08.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:08.206 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:08.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:08.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:08.337 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:08.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:08.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:08.480 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:08.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:08.582 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:08.609 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:08.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:08.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:08.749 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:08.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:08.764 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:40:08.769 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 11 pgp_num to 16 2024-03-24T11:40:08.852 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:08.879 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:08.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:08 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/635429640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:08 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3340995219' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-24T11:40:08.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:08 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-24T11:40:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:08 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/635429640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:08 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3340995219' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-24T11:40:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:08 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-24T11:40:08.981 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:09.025 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:09.133 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:08 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/635429640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:08 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3340995219' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-24T11:40:09.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:08 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-24T11:40:09.172 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:09.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.304 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:09.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:09.405 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:09.444 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:09.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:09.574 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:09.676 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.715 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:09.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:09.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.841 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:09.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:09 smithi181 ceph-mon[33998]: pgmap v984: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 699 op/s; 133/87105 objects misplaced (0.153%); 3.1 MiB/s, 48 objects/s recovering 2024-03-24T11:40:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:09 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/635429640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:09 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-24T11:40:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:09 smithi181 ceph-mon[33998]: osdmap e447: 8 total, 8 up, 6 in 2024-03-24T11:40:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:09 smithi181 ceph-mon[33998]: 11.1 scrub starts 2024-03-24T11:40:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:09 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:40:09.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:09.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:09.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:10.072 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:10.116 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:10.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[29608]: pgmap v984: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 699 op/s; 133/87105 objects misplaced (0.153%); 3.1 MiB/s, 48 objects/s recovering 2024-03-24T11:40:10.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:10.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/635429640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:10.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-24T11:40:10.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[29608]: osdmap e447: 8 total, 8 up, 6 in 2024-03-24T11:40:10.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[29608]: 11.1 scrub starts 2024-03-24T11:40:10.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:40:10.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[37907]: pgmap v984: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 699 op/s; 133/87105 objects misplaced (0.153%); 3.1 MiB/s, 48 objects/s recovering 2024-03-24T11:40:10.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:10.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/635429640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:10.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-24T11:40:10.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[37907]: osdmap e447: 8 total, 8 up, 6 in 2024-03-24T11:40:10.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[37907]: 11.1 scrub starts 2024-03-24T11:40:10.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:09 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:40:10.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:10.289 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:10.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:10.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:10.435 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:10.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:10.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:10.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:10.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:10.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:10.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:10.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:10.832 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:10.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:10.880 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[29608]: 11.1 scrub ok 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[29608]: osdmap e448: 8 total, 8 up, 6 in 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[37907]: 11.1 scrub ok 2024-03-24T11:40:10.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:10 smithi179 ceph-mon[37907]: osdmap e448: 8 total, 8 up, 6 in 2024-03-24T11:40:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:10 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:40:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:10 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:40:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:10 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:40:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:10 smithi181 ceph-mon[33998]: 11.1 scrub ok 2024-03-24T11:40:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:10 smithi181 ceph-mon[33998]: osdmap e448: 8 total, 8 up, 6 in 2024-03-24T11:40:10.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:10.988 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:10.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:11.090 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:11.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:11.132 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:11.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:11.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:11.270 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:11.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:11.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:11.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:11.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:11.530 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:11.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:11.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:11.658 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:11.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:11.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:11.787 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:11.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:11.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:11.914 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:11.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 11.b scrub starts 2024-03-24T11:40:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: pgmap v987: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 982 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-03-24T11:40:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 11.b scrub ok 2024-03-24T11:40:11.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 11.b scrub starts 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: pgmap v987: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 982 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 11.b scrub ok 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:40:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:40:11.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:40:11.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:11 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:40:12.004 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:40:12.017 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:12.054 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:40:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:12.065 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:12.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:12.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 11.b scrub starts 2024-03-24T11:40:12.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: pgmap v987: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 982 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-03-24T11:40:12.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 11.b scrub ok 2024-03-24T11:40:12.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:40:12.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:40:12.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:40:12.100 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:40:12.100 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:40:12.100 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:11 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:40:12.110 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:40:12.152 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:40:12.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:12.192 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:40:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:12.212 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:12.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:12.239 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:40:12.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:40:12.281 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:40:12.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:12.318 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:40:12.324 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:40:12.324 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:40:12.324 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:40:12.324 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:40:12.324 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:40:12.324 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:40:12.324 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:40:12.325 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:40:12.342 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:12.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:12.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:12.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:12.531 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:12.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:12.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:12.688 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:12.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:12.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:12.819 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:12.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:12.834 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:12.834 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:12.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:12.945 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:12.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: 11.9 scrub starts 2024-03-24T11:40:12.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: 11.9 scrub ok 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3738789809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: 11.9 scrub starts 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: 11.9 scrub ok 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3738789809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:12.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:40:12.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:12 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:40:13.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:13.072 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: 11.9 scrub starts 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: 11.9 scrub ok 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3738789809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:40:13.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:12 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:40:13.168 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:13.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:13.201 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:13.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:13.329 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:13.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:13.472 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:13.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:13.600 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:13.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:13.728 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:13.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.858 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:13.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:13.959 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:13.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[29608]: 4.14 scrub starts 2024-03-24T11:40:13.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[29608]: 4.14 scrub ok 2024-03-24T11:40:13.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[29608]: pgmap v988: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 777 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-24T11:40:13.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[29608]: 1.2 scrub starts 2024-03-24T11:40:13.962 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[29608]: 1.2 scrub ok 2024-03-24T11:40:13.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[37907]: 4.14 scrub starts 2024-03-24T11:40:13.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[37907]: 4.14 scrub ok 2024-03-24T11:40:13.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[37907]: pgmap v988: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 777 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-24T11:40:13.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[37907]: 1.2 scrub starts 2024-03-24T11:40:13.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:13 smithi179 ceph-mon[37907]: 1.2 scrub ok 2024-03-24T11:40:13.987 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:13.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:14.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:14.118 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:14.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:13 smithi181 ceph-mon[33998]: 4.14 scrub starts 2024-03-24T11:40:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:13 smithi181 ceph-mon[33998]: 4.14 scrub ok 2024-03-24T11:40:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:13 smithi181 ceph-mon[33998]: pgmap v988: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 777 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-24T11:40:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:13 smithi181 ceph-mon[33998]: 1.2 scrub starts 2024-03-24T11:40:14.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:13 smithi181 ceph-mon[33998]: 1.2 scrub ok 2024-03-24T11:40:14.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:14.266 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:14.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:14.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:14.410 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:14.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:14.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:14.596 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:14.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:14.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:40:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:40:14.696 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:14.739 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:14.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:14.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:14.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:14.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:14.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:14.945 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T11:40:14.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.013 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:15.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:15.050 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[29608]: 6.2 scrub starts 2024-03-24T11:40:15.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[29608]: 6.2 scrub ok 2024-03-24T11:40:15.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:40:15.051 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:40:15.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[37907]: 6.2 scrub starts 2024-03-24T11:40:15.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[37907]: 6.2 scrub ok 2024-03-24T11:40:15.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:40:15.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:14 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:40:15.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:40:15.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:15.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:14 smithi181 ceph-mon[33998]: 6.2 scrub starts 2024-03-24T11:40:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:14 smithi181 ceph-mon[33998]: 6.2 scrub ok 2024-03-24T11:40:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:14 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:40:15.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:14 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:40:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.183 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:15.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:15.251 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:15.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:15.311 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:15.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:15.441 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:15.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:40:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:15.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:15.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:15.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:15.702 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:15.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.833 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:15.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:15.934 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:15.978 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:15.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:16.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:16.107 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:16.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:16.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:16.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: pgmap v989: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s; 1.0 MiB/s, 16 objects/s recovering 2024-03-24T11:40:16.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:16.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:40:16.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:16.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:40:16.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:40:16.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:40:16.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:15 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:40:16.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:16.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: pgmap v989: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s; 1.0 MiB/s, 16 objects/s recovering 2024-03-24T11:40:16.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:16.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:40:16.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:16.171 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: pgmap v989: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s; 1.0 MiB/s, 16 objects/s recovering 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:40:16.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:15 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:40:16.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:16.235 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:16.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:16.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:16.366 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:16.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:16.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6] out_osds: [4, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T11:40:16.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T11:40:16.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:16.494 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:16.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:16.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:16.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:16.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:16.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:16.746 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:16.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:16.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:16.872 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:16.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:16.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:17.019 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:17.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 3.b scrub starts 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 3.b scrub ok 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:40:17.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:40:17.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:40:17.074 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:16 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:40:17.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:17.148 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:17.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:17.186 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:17.204 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 3.b scrub starts 2024-03-24T11:40:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 3.b scrub ok 2024-03-24T11:40:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:40:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:40:17.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 3.b scrub starts 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 3.b scrub ok 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:40:17.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:40:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:40:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:40:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:40:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:40:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:40:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:16 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:40:17.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:17.299 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:17.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:17.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:17.428 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:40:17.429 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:40:17.429 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:40:17.429 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:40:17.429 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:40:17.430 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:17.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:17.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:17.572 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:17.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:17.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:17.706 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:17.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:17.807 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:17.826 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:40:17.858 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:17.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:17.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: pgmap v990: 158 pgs: 158 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 1016 KiB/s, 15 objects/s recovering 2024-03-24T11:40:17.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: 11.c scrub starts 2024-03-24T11:40:17.938 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1380515273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: pgmap v990: 158 pgs: 158 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 1016 KiB/s, 15 objects/s recovering 2024-03-24T11:40:17.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: 11.c scrub starts 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1380515273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:17.940 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:17 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:17.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:18.006 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:18.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:18.137 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:18.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: pgmap v990: 158 pgs: 158 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 1016 KiB/s, 15 objects/s recovering 2024-03-24T11:40:18.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: 11.c scrub starts 2024-03-24T11:40:18.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:40:18.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:40:18.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:40:18.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:40:18.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:40:18.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:40:18.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1380515273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:18.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:17 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:18.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T11:40:18.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:18.296 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:18.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:18.426 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:18.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:18.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:18.531 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.562 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:18.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:18.663 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.701 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:18.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:18.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:18.833 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:18.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: 6.c scrub starts 2024-03-24T11:40:18.924 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: 6.c scrub ok 2024-03-24T11:40:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: 11.c scrub ok 2024-03-24T11:40:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:40:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:40:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:40:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: osdmap e449: 8 total, 8 up, 6 in 2024-03-24T11:40:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:40:18.925 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:18 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:40:18.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:18.964 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:18.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:19.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:19.094 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:19.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:19.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:19.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: 6.c scrub starts 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: 6.c scrub ok 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: 11.c scrub ok 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: osdmap e449: 8 total, 8 up, 6 in 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: 6.c scrub starts 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: 6.c scrub ok 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: 11.c scrub ok 2024-03-24T11:40:19.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:40:19.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:40:19.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:40:19.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: osdmap e449: 8 total, 8 up, 6 in 2024-03-24T11:40:19.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:40:19.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:18 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:40:19.232 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:19.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:19.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:19.360 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:19.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:19.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:19.490 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:19.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:19.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:19.633 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:19.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:19.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:19.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:19.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:19.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:19.892 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:19.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:19.994 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:20.037 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:20.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:20.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:19 smithi181 ceph-mon[33998]: pgmap v992: 158 pgs: 158 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 587 B/s rd, 0 B/s wr, 774 op/s 2024-03-24T11:40:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:19 smithi181 ceph-mon[33998]: 5.c deep-scrub starts 2024-03-24T11:40:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:19 smithi181 ceph-mon[33998]: 5.c deep-scrub ok 2024-03-24T11:40:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:20.177 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:20.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[29608]: pgmap v992: 158 pgs: 158 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 587 B/s rd, 0 B/s wr, 774 op/s 2024-03-24T11:40:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[29608]: 5.c deep-scrub starts 2024-03-24T11:40:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[29608]: 5.c deep-scrub ok 2024-03-24T11:40:20.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:20.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[37907]: pgmap v992: 158 pgs: 158 active+clean; 1.2 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 587 B/s rd, 0 B/s wr, 774 op/s 2024-03-24T11:40:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[37907]: 5.c deep-scrub starts 2024-03-24T11:40:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[37907]: 5.c deep-scrub ok 2024-03-24T11:40:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:20.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:20.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:20.314 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:20.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:20.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:20.442 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:20.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:20.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:20.578 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:20.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:20.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:20.724 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:20.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:20.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:20.850 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213990 10.c 10:33f1c950:::benchmark_data_smithi179_155357_object103494:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.291870+0000", 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 235.556402967, 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.065501571, 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:20.851 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213990 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291870+0000", 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291870+0000", 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291872+0000", 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7579999999999999e-06 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291881+0000", 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3889999999999994e-06 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291894+0000", 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2904e-05 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291904+0000", 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7440000000000002e-06 2024-03-24T11:40:20.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.293526+0000", 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001621707 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.293531+0000", 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6710000000000004e-06 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.503629+0000", 2024-03-24T11:40:20.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.210097325 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.503643+0000", 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3916e-05 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.357287+0000", 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.85364431699999999 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.357372+0000", 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4839999999999994e-05 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213992 10.1c 10:3e0f01f5:::benchmark_data_smithi179_155357_object103496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.298999+0000", 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 235.54927451200001, 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1072013119999999, 2024-03-24T11:40:20.857 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213992 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.298999+0000", 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.298999+0000", 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299001+0000", 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3089999999999998e-06 2024-03-24T11:40:20.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299008+0000", 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5530000000000004e-06 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299022+0000", 2024-03-24T11:40:20.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3888e-05 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.299032+0000", 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3309999999999993e-06 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504217+0000", 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20518531300000001 2024-03-24T11:40:20.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504224+0000", 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0049999999999998e-06 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.382915+0000", 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.87869146499999995 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.383056+0000", 2024-03-24T11:40:20.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014087600000000001 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.383291+0000", 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000234916 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387031+0000", 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037401560000000001 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387043+0000", 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1688000000000001e-05 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.406147+0000", 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019103983000000001 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.406177+0000", 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0324999999999998e-05 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.406200+0000", 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2504e-05 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213999 10.1c 10:3df129a2:::benchmark_data_smithi179_155357_object103503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.341073+0000", 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 235.507199679, 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.067161144, 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.865 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213999 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341073+0000", 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341073+0000", 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341076+0000", 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5610000000000001e-06 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341086+0000", 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9930000000000007e-06 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341116+0000", 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0023999999999999e-05 2024-03-24T11:40:20.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.341132+0000", 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6079999999999999e-05 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504245+0000", 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.163112966 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.504251+0000", 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5860000000000004e-06 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.385818+0000", 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.88156710900000002 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.385938+0000", 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120684 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.386116+0000", 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017782799999999999 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.389597+0000", 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034809900000000002 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.389605+0000", 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2690000000000002e-06 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408134+0000", 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018529747999999999 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408190+0000", 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6209999999999999e-05 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408235+0000", 2024-03-24T11:40:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4095999999999999e-05 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214003 10.21 10:84cdce02:::benchmark_data_smithi179_155357_object103507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.544295+0000", 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 235.30397823300001, 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82516384700000001, 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214003 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544295+0000", 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544295+0000", 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544297+0000", 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8789999999999999e-06 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544304+0000", 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3440000000000003e-06 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544326+0000", 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1441999999999999e-05 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544338+0000", 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2578999999999999e-05 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544423+0000", 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4643999999999998e-05 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.544436+0000", 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2926e-05 2024-03-24T11:40:20.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360241+0000", 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81580506799999997 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360368+0000", 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000127272 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360615+0000", 2024-03-24T11:40:20.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024656500000000002 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.363504+0000", 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002889015 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.363650+0000", 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014628300000000001 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369348+0000", 2024-03-24T11:40:20.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005698196 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369410+0000", 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1618000000000005e-05 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369459+0000", 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9016e-05 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.881 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214004 10.1c 10:3bbddca2:::benchmark_data_smithi179_155357_object103508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.553160+0000", 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 235.29511341700001, 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.85520687100000004, 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214004 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.882 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553160+0000", 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553160+0000", 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553161+0000", 2024-03-24T11:40:20.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3829999999999999e-06 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553166+0000", 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1560000000000003e-06 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553179+0000", 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.325e-05 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553188+0000", 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7269999999999996e-06 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553256+0000", 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8279000000000001e-05 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.553267+0000", 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0647e-05 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387067+0000", 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83380036400000002 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387194+0000", 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126978 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,7", 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.387331+0000", 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013626799999999999 2024-03-24T11:40:20.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.389607+0000", 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022766819999999999 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.390696+0000", 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010887539999999999 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408280+0000", 2024-03-24T11:40:20.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017583863000000002 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408329+0000", 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9282e-05 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.408367+0000", 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7237999999999998e-05 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215487 10.23 10.e5a750a3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.360194+0000", 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 233.48807890699999, 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96463855099999996, 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215487 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360194+0000", 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360194+0000", 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360195+0000", 2024-03-24T11:40:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2139999999999999e-06 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360201+0000", 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4500000000000003e-06 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360211+0000", 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0478e-05 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360217+0000", 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9109999999999998e-06 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360284+0000", 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6818999999999998e-05 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360294+0000", 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0278000000000001e-05 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.324822+0000", 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96452802900000001 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.324833+0000", 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0372e-05 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215490 10.21 10:84f9e374:::benchmark_data_smithi179_155357_object104994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.360982+0000", 2024-03-24T11:40:20.893 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 233.48729105499999, 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.070579167, 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215490 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360982+0000", 2024-03-24T11:40:20.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360982+0000", 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360983+0000", 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5400000000000001e-07 2024-03-24T11:40:20.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360986+0000", 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.613e-06 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.360996+0000", 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5400000000000001e-06 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.361002+0000", 2024-03-24T11:40:20.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.66e-06 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.361062+0000", 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0346999999999997e-05 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.361075+0000", 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2673e-05 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.320218+0000", 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95914328800000004 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.320234+0000", 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6316000000000001e-05 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.419190+0000", 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.098956238 2024-03-24T11:40:20.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.419341+0000", 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015083699999999999 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,2,7", 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.419576+0000", 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023469699999999999 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.420780+0000", 2024-03-24T11:40:20.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001203976 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.422186+0000", 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014063859999999999 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.425236+0000", 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030498109999999999 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.431359+0000", 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061229530000000004 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.431439+0000", 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9555000000000007e-05 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.431561+0000", 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012262300000000001 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215525 10.1d 10:bd921279:::benchmark_data_smithi179_155357_object105029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.407932+0000", 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 233.44034080599999, 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0504010400000001, 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215525 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.902 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407932+0000", 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407932+0000", 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407933+0000", 2024-03-24T11:40:20.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0860000000000001e-06 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407938+0000", 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.25e-06 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407948+0000", 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0584999999999999e-05 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.407957+0000", 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3070000000000003e-06 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.408014+0000", 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7741000000000002e-05 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.408022+0000", 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8089999999999993e-06 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323440+0000", 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91541776699999999 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323452+0000", 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1759999999999999e-05 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.450011+0000", 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1265593039999999 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.450121+0000", 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109869 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6", 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.450362+0000", 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024074900000000001 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451902+0000", 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015406549999999999 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458058+0000", 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061556670000000001 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458108+0000", 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0351e-05 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458161+0000", 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3297000000000001e-05 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458224+0000", 2024-03-24T11:40:20.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2209e-05 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458333+0000", 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109634 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215434 10.1c 10:3e049106:::benchmark_data_smithi179_155357_object104938:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.322005+0000", 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 232.526267889, 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.119395546, 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.911 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215434 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322005+0000", 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322005+0000", 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322007+0000", 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9879999999999999e-06 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322012+0000", 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6700000000000002e-06 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322925+0000", 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091357500000000002 2024-03-24T11:40:20.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322939+0000", 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4015e-05 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325695+0000", 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002755118 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325704+0000", 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7499999999999998e-06 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441300+0000", 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.115596145 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441401+0000", 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010028500000000001 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.915 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215438 10.1c 10:3d6c033a:::benchmark_data_smithi179_155357_object104942:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325611+0000", 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 232.52266203799999, 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.13719357, 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.916 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215438 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325611+0000", 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325611+0000", 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325612+0000", 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2160000000000001e-06 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325618+0000", 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6280000000000002e-06 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325632+0000", 2024-03-24T11:40:20.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4453999999999999e-05 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325643+0000", 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0468999999999999e-05 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325784+0000", 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014096700000000001 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325792+0000", 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8520000000000004e-06 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441694+0000", 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.115902816 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441826+0000", 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013180399999999999 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.462624+0000", 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.020798179999999999 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.462805+0000", 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018018400000000001 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215480 10.1c 10:3cfdbf27:::benchmark_data_smithi179_155357_object104984:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325659+0000", 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 232.522614205, 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.125743108, 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.923 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215480 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325659+0000", 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325659+0000", 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325660+0000", 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3e-07 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325663+0000", 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4410000000000002e-06 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325671+0000", 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2800000000000003e-06 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325678+0000", 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.041e-06 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325818+0000", 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013960200000000001 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325825+0000", 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2760000000000003e-06 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.441915+0000", 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1160902100000001 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.928 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:20.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:20.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.442023+0000", 2024-03-24T11:40:20.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010732 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6,7", 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.442240+0000", 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000217004 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447810+0000", 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055700369999999999 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447934+0000", 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012472699999999999 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447971+0000", 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6705999999999997e-05 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.448006+0000", 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5228999999999998e-05 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451173+0000", 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003166678 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451270+0000", 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6434999999999994e-05 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451402+0000", 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000132492 2024-03-24T11:40:20.932 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215504 10.1c 10:3d911fcf:::benchmark_data_smithi179_155357_object105008:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 232.52258531300001, 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.129241637, 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.933 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215504 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325688+0000", 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7199999999999999e-07 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325691+0000", 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9189999999999999e-06 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325699+0000", 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4359999999999996e-06 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325706+0000", 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9110000000000004e-06 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325853+0000", 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014751599999999999 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325861+0000", 2024-03-24T11:40:20.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7319999999999998e-06 2024-03-24T11:40:20.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443267+0000", 2024-03-24T11:40:20.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1174060210000001 2024-03-24T11:40:20.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443382+0000", 2024-03-24T11:40:20.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000115062 2024-03-24T11:40:20.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6,7", 2024-03-24T11:40:20.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443550+0000", 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016815499999999999 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447853+0000", 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043024960000000003 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.448043+0000", 2024-03-24T11:40:20.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000190173 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451443+0000", 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003400368 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454549+0000", 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003105642 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454761+0000", 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021172199999999999 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454813+0000", 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.253e-05 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454929+0000", 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116382 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.942 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215517 10.1c 10:39423bd9:::benchmark_data_smithi179_155357_object105021:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.325817+0000", 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 232.522455721, 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1324648829999999, 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215517 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.943 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325817+0000", 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325817+0000", 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325818+0000", 2024-03-24T11:40:20.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4600000000000004e-07 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325823+0000", 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3869999999999997e-06 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325834+0000", 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1126e-05 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325842+0000", 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.507e-06 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325887+0000", 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4697000000000003e-05 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.325894+0000", 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8600000000000004e-06 2024-03-24T11:40:20.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446112+0000", 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.120217829 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446229+0000", 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117335 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1,6,7", 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446435+0000", 2024-03-24T11:40:20.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000205859 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.451868+0000", 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054331620000000001 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454562+0000", 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026939120000000001 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454719+0000", 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015704400000000001 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.454965+0000", 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000245937 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.455045+0000", 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0037000000000006e-05 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458134+0000", 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030891690000000001 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.458282+0000", 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000148276 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217810 10.1d 10:b9e8646c:::benchmark_data_smithi179_155357_object107314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:40:20.950 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.390708+0000", 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.457564977, 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98764608099999995, 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217810 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390708+0000", 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390708+0000", 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390709+0000", 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8899999999999999e-07 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390713+0000", 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3579999999999996e-06 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390732+0000", 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8569000000000001e-05 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390738+0000", 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8710000000000002e-06 2024-03-24T11:40:20.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390789+0000", 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1566000000000001e-05 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.390798+0000", 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0119999999999994e-06 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.374933+0000", 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98413457699999995 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.375047+0000", 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011410599999999999 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.375249+0000", 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000202258 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378145+0000", 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028957010000000001 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378196+0000", 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1397000000000003e-05 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378260+0000", 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3486999999999997e-05 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378313+0000", 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3588999999999998e-05 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378354+0000", 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0901e-05 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217862 10.1d 10:ba4afeec:::benchmark_data_smithi179_155357_object107366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:40:20.957 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.430683+0000", 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.41759050799999, 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.94900074700000003, 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217862 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430683+0000", 2024-03-24T11:40:20.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430683+0000", 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430684+0000", 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1060000000000001e-06 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430688+0000", 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6809999999999997e-06 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430699+0000", 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0302000000000001e-05 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430705+0000", 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0440000000000001e-06 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430763+0000", 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8712000000000003e-05 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.430774+0000", 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0396e-05 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.376471+0000", 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.94569716100000001 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.376549+0000", 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7528999999999996e-05 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.376708+0000", 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000159922 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378426+0000", 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017170810000000001 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379509+0000", 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010837690000000001 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379583+0000", 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3565999999999993e-05 2024-03-24T11:40:20.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379640+0000", 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6780000000000002e-05 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379683+0000", 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3698000000000001e-05 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217905 10.1d 10:bd552d80:::benchmark_data_smithi179_155357_object107409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.389041585, 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.92063982300000002, 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217905 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459232+0000", 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-03-24T11:40:20.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459234+0000", 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9010000000000001e-06 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459238+0000", 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5589999999999999e-06 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459242+0000", 2024-03-24T11:40:20.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3639999999999999e-06 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459287+0000", 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4901000000000003e-05 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.459295+0000", 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8759999999999994e-06 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.377420+0000", 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91812414499999995 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.377483+0000", 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3854000000000002e-05 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.377593+0000", 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109219 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379524+0000", 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019309500000000001 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379739+0000", 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021520099999999999 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379780+0000", 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0915999999999999e-05 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379831+0000", 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0844000000000003e-05 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379871+0000", 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0757000000000003e-05 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.971 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217927 10.1d 10:bcda50cf:::benchmark_data_smithi179_155357_object107431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.476986+0000", 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.37128703900001, 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90311201900000004, 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217927 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476986+0000", 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476986+0000", 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476988+0000", 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.502e-06 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.476994+0000", 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6069999999999996e-06 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477016+0000", 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1611e-05 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477024+0000", 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6910000000000003e-06 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477077+0000", 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2475999999999997e-05 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.477083+0000", 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.297e-06 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378453+0000", 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90136997900000004 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378556+0000", 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103014 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.378686+0000", 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012929 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379933+0000", 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001247791 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.379984+0000", 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1115000000000001e-05 2024-03-24T11:40:20.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380003+0000", 2024-03-24T11:40:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8706999999999999e-05 2024-03-24T11:40:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380053+0000", 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9873000000000002e-05 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380098+0000", 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5065999999999998e-05 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217957 10.1d 10:ba42563d:::benchmark_data_smithi179_155357_object107461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:40:20.979 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.497518+0000", 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.35075490599999, 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.89107709099999999, 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217957 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497518+0000", 2024-03-24T11:40:20.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497518+0000", 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497519+0000", 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6300000000000005e-07 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497523+0000", 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7960000000000002e-06 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497532+0000", 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.781e-06 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497540+0000", 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.323e-06 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497628+0000", 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8265999999999998e-05 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.497638+0000", 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0192e-05 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380134+0000", 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.88249543799999997 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380244+0000", 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011079 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.380416+0000", 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171853 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.383884+0000", 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003468151 2024-03-24T11:40:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.387184+0000", 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032990710000000002 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388495+0000", 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001311638 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388555+0000", 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0272999999999997e-05 2024-03-24T11:40:20.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388595+0000", 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9855999999999999e-05 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:217989 10.1d 10:bd18519e:::benchmark_data_smithi179_155357_object107493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.521620+0000", 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.326652927, 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.867147318, 2024-03-24T11:40:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 217989 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521620+0000", 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521620+0000", 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521621+0000", 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7899999999999998e-07 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521624+0000", 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3639999999999999e-06 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521633+0000", 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3510000000000002e-06 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521639+0000", 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.545e-06 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521698+0000", 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8452999999999998e-05 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.521707+0000", 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5540000000000003e-06 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.381365+0000", 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.85965779099999995 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.381469+0000", 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104182 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.381640+0000", 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000170636 2024-03-24T11:40:20.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.383890+0000", 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022498219999999998 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.387314+0000", 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034244560000000002 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388662+0000", 2024-03-24T11:40:20.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001348094 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388718+0000", 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5823999999999999e-05 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388767+0000", 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9567000000000002e-05 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:20.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:218013 10.1d 10:be135f22:::benchmark_data_smithi179_155357_object107517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:31.547096+0000", 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 229.30117704899999, 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.84178786999999999, 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 218013 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.994 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547096+0000", 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547096+0000", 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547097+0000", 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0499999999999999e-06 2024-03-24T11:40:20.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547102+0000", 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4460000000000003e-06 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547114+0000", 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2875e-05 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547122+0000", 2024-03-24T11:40:20.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7859999999999997e-06 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547172+0000", 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9935999999999998e-05 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:31.547178+0000", 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5600000000000001e-06 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:20.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.382589+0000", 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83541153800000001 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.382696+0000", 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106716 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,6", 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.382884+0000", 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000187596 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.385085+0000", 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022019309999999999 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.387360+0000", 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002274781 2024-03-24T11:40:20.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388810+0000", 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014493189999999999 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388851+0000", 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1108e-05 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:32.388884+0000", 2024-03-24T11:40:21.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3228000000000001e-05 2024-03-24T11:40:21.001 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:21.001 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:21.001 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:21.001 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:21.001 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:21.001 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:40:21.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:21.058 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:21.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:21.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:21.200 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:21.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:21.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:21.341 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:21.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:21.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:21.470 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:21.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:21.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:21.598 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:21.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:21.700 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:21.744 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:21.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:21.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:21.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T11:40:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:21.875 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:21.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:21.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:22.025 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:22.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 5.17 scrub starts 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 5.17 scrub ok 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: pgmap v993: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 831 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:40:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:21 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:40:22.155 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:22.156 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 5.17 scrub starts 2024-03-24T11:40:22.156 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 5.17 scrub ok 2024-03-24T11:40:22.156 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: pgmap v993: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 831 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:40:22.156 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:40:22.156 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:40:22.156 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:40:22.156 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 5.17 scrub starts 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 5.17 scrub ok 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: pgmap v993: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 831 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:40:22.157 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:40:22.158 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:21 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:40:22.177 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:22.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:22.290 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:40:22.306 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:22.338 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:40:22.383 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:40:22.410 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:22.431 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:40:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.450 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:22.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:22.471 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:40:22.509 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:40:22.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:22.576 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:22.606 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:40:22.650 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:40:22.655 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:40:22.655 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:40:22.656 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:40:22.656 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:40:22.656 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:40:22.656 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:40:22.656 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:40:22.656 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:40:22.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:22.705 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:22.807 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:22.844 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:22.946 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:22.984 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:22.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:23.086 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:23.124 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:23.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:23.148 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4240266160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:23.148 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[29608]: 4.7 deep-scrub starts 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[29608]: 4.7 deep-scrub ok 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4240266160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[37907]: 4.7 deep-scrub starts 2024-03-24T11:40:23.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:22 smithi179 ceph-mon[37907]: 4.7 deep-scrub ok 2024-03-24T11:40:23.151 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:22 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4240266160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:22 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:40:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:22 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:40:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:22 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:40:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:22 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:40:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:22 smithi181 ceph-mon[33998]: 4.7 deep-scrub starts 2024-03-24T11:40:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:22 smithi181 ceph-mon[33998]: 4.7 deep-scrub ok 2024-03-24T11:40:23.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:23.252 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:23.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:23.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:23.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T11:40:23.381 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:23.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:23.483 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:23.539 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:23.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:23.640 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:23.690 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:23.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:23.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:23.829 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:23.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:23.902 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T11:40:23.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:23.960 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:23.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:23.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:23 smithi181 ceph-mon[33998]: pgmap v994: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 831 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:40:23.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3786039571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:40:23.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:40:23.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:23 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:40:23.963 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:23 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:40:24.025 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:24.025 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:24.062 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:24.092 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:24.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:24.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[29608]: pgmap v994: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 831 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:40:24.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3786039571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:40:24.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:40:24.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:40:24.144 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:40:24.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[37907]: pgmap v994: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 831 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T11:40:24.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3786039571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:40:24.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:40:24.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:40:24.145 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:23 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:40:24.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:24.245 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:24.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:24.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:24.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:24.376 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:24.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:24.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:24.516 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:24.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:24.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:24.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:40:24.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:24.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:24.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T11:40:24.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:24.819 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:24.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:24.921 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:24.965 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:24.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:24.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T11:40:25.032 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:25.055 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:25.067 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:25.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:25.117 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:25.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:40:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-mon[33998]: osdmap e450: 8 total, 8 up, 6 in 2024-03-24T11:40:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-mon[33998]: 4.11 scrub starts 2024-03-24T11:40:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-mon[33998]: 4.11 scrub ok 2024-03-24T11:40:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:40:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:40:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:24 smithi181 ceph-mon[33998]: 6.15 deep-scrub starts 2024-03-24T11:40:25.183 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:40:25.184 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[29608]: osdmap e450: 8 total, 8 up, 6 in 2024-03-24T11:40:25.184 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[29608]: 4.11 scrub starts 2024-03-24T11:40:25.184 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[29608]: 4.11 scrub ok 2024-03-24T11:40:25.184 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:40:25.184 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:40:25.184 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[29608]: 6.15 deep-scrub starts 2024-03-24T11:40:25.184 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:40:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:25.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:40:25.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[37907]: osdmap e450: 8 total, 8 up, 6 in 2024-03-24T11:40:25.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[37907]: 4.11 scrub starts 2024-03-24T11:40:25.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[37907]: 4.11 scrub ok 2024-03-24T11:40:25.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:40:25.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:40:25.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:24 smithi179 ceph-mon[37907]: 6.15 deep-scrub starts 2024-03-24T11:40:25.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:25.276 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:25.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:25.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:25.409 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:25.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:25.511 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:25.538 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:25.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:25.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:25.667 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:25.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:25.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:25.812 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:25.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:25.914 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T11:40:25.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:25.942 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:25.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:26.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:26.071 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:26.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:25 smithi181 ceph-mon[33998]: 6.15 deep-scrub ok 2024-03-24T11:40:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:25 smithi181 ceph-mon[33998]: pgmap v996: 158 pgs: 158 active+clean; 599 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:40:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:25 smithi181 ceph-mon[33998]: 3.1f deep-scrub starts 2024-03-24T11:40:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:25 smithi181 ceph-mon[33998]: 3.1f deep-scrub ok 2024-03-24T11:40:26.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:26.199 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:26.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[29608]: 6.15 deep-scrub ok 2024-03-24T11:40:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[29608]: pgmap v996: 158 pgs: 158 active+clean; 599 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:40:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[29608]: 3.1f deep-scrub starts 2024-03-24T11:40:26.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[29608]: 3.1f deep-scrub ok 2024-03-24T11:40:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[37907]: 6.15 deep-scrub ok 2024-03-24T11:40:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[37907]: pgmap v996: 158 pgs: 158 active+clean; 599 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T11:40:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[37907]: 3.1f deep-scrub starts 2024-03-24T11:40:26.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:25 smithi179 ceph-mon[37907]: 3.1f deep-scrub ok 2024-03-24T11:40:26.301 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:26.327 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:26.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:26.428 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:26.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:26.447 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:26.500 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:26.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:26.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:26.628 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:26.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:26.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:26.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:26.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:26.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:26.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:26.908 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:26.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:27.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:27.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:40:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:40:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 6.b scrub starts 2024-03-24T11:40:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:40:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:40:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:40:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:40:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:26 smithi181 ceph-mon[33998]: 6.b scrub ok 2024-03-24T11:40:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.039 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:27.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:27.092 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:40:27.092 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 6.b scrub starts 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[29608]: 6.b scrub ok 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 6.b scrub starts 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:40:27.093 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:40:27.094 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:40:27.094 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:40:27.094 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:26 smithi179 ceph-mon[37907]: 6.b scrub ok 2024-03-24T11:40:27.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:27.166 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:27.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:27.294 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:27.310 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:27.310 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:27.397 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:27.440 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:27.542 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.569 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:27.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:27.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:27.697 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:27.801 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:27.841 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:27.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:27.952 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is set 2024-03-24T11:40:27.968 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:27.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:28.037 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:27 smithi179 ceph-mon[29608]: pgmap v997: 158 pgs: 158 active+clean; 491 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-24T11:40:28.037 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:27 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1054338765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:40:28.037 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:27 smithi179 ceph-mon[37907]: pgmap v997: 158 pgs: 158 active+clean; 491 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-24T11:40:28.037 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:27 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1054338765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:40:28.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:28.104 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:28.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:28.116 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T11:40:28.185 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:27 smithi181 ceph-mon[33998]: pgmap v997: 158 pgs: 158 active+clean; 491 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-24T11:40:28.185 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:27 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1054338765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T11:40:28.207 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:28.244 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:28.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:28.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:28.378 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:28.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:28.456 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:28.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:28.519 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:28.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:28.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:28.651 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:28.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:28.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:28.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:28.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:28.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:28.921 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:28.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:29.023 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:29.051 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:29.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:28 smithi181 ceph-mon[33998]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1054338765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:40:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:28 smithi181 ceph-mon[33998]: osdmap e451: 8 total, 8 up, 6 in 2024-03-24T11:40:29.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:29.191 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:29.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:29.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:28 smithi179 ceph-mon[29608]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1054338765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:40:29.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:28 smithi179 ceph-mon[29608]: osdmap e451: 8 total, 8 up, 6 in 2024-03-24T11:40:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:28 smithi179 ceph-mon[37907]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1054338765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T11:40:29.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:28 smithi179 ceph-mon[37907]: osdmap e451: 8 total, 8 up, 6 in 2024-03-24T11:40:29.293 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:29.335 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:29.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:29.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:29.464 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:29.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:29.566 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:29.590 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:29.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:29.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:29.728 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:29.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:29.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:29.871 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:29.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:29.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:30.011 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:30.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:29 smithi181 ceph-mon[33998]: pgmap v999: 158 pgs: 158 active+clean; 491 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 681 op/s 2024-03-24T11:40:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.153 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:30.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:30.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:29 smithi179 ceph-mon[29608]: pgmap v999: 158 pgs: 158 active+clean; 491 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 681 op/s 2024-03-24T11:40:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:29 smithi179 ceph-mon[37907]: pgmap v999: 158 pgs: 158 active+clean; 491 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 681 op/s 2024-03-24T11:40:30.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:30.285 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:30.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:30.288 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:30.289 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:30.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:30.414 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:30.516 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:30.557 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:30.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:30.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:30.689 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:30.791 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:30.832 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:30.902 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:40:30 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a[64806]: logger=cleanup t=2024-03-24T11:40:30.494736131Z level=info msg="Completed cleanup jobs" duration=13.935403ms 2024-03-24T11:40:30.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:30.934 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Removed 120295 objects 2024-03-24T11:40:30.934 INFO:tasks.radosbench.radosbench.0.smithi179.stdout:Clean up completed and total clean up time :117.767 2024-03-24T11:40:30.960 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:30.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:31.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.087 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:31.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:31.188 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:30 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:40:31.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:30 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:40:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:30 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:40:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:30 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:40:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.213 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:31.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:31.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.340 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:31.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:31.393 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:30 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:40:31.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:30 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:40:31.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T11:40:31.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.488 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:31.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:31.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:31.615 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:31.664 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:31.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.741 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:31.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:31.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.868 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:31.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:31.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:31.994 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:31.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:32.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:31 smithi181 ceph-mon[33998]: pgmap v1000: 158 pgs: 158 active+clean; 127 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-24T11:40:32.039 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:31 smithi181 ceph-mon[33998]: osdmap e452: 8 total, 8 up, 6 in 2024-03-24T11:40:32.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:32.108 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:31 smithi179 ceph-mon[29608]: pgmap v1000: 158 pgs: 158 active+clean; 127 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-24T11:40:32.108 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:31 smithi179 ceph-mon[29608]: osdmap e452: 8 total, 8 up, 6 in 2024-03-24T11:40:32.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:31 smithi179 ceph-mon[37907]: pgmap v1000: 158 pgs: 158 active+clean; 127 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-24T11:40:32.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:31 smithi179 ceph-mon[37907]: osdmap e452: 8 total, 8 up, 6 in 2024-03-24T11:40:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:32.120 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:32.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:32.221 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:32.246 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:32.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:32.348 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:32.372 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:32.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:32.474 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:32.498 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:32.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:32.599 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:32.623 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:32.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:32.725 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:32.748 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:32.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:32.850 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:32.875 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:32.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:32.961 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T11:40:32.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.000 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:33.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:33.088 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:32 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4149891062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:40:33.089 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:32 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:40:33.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:33.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:32 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4149891062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:40:33.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:32 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:40:33.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:32 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4149891062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:40:33.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:32 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:40:33.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:33.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.250 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:33.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:33.329 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:33.330 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:33.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.376 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:33.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:33.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:33.501 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:33.603 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:33.629 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:33.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:33.757 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:33.798 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T11:40:33.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:33.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:33.987 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:34.010 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:34.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:34.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:34.136 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: pgmap v1002: 158 pgs: 158 active+clean; 127 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 943 op/s 2024-03-24T11:40:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:40:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: osdmap e453: 8 total, 8 up, 6 in 2024-03-24T11:40:34.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]: dispatch 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]: dispatch 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]: dispatch 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]: dispatch 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: pgmap v1002: 158 pgs: 158 active+clean; 127 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 943 op/s 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: osdmap e453: 8 total, 8 up, 6 in 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]: dispatch 2024-03-24T11:40:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]: dispatch 2024-03-24T11:40:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-03-24T11:40:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]: dispatch 2024-03-24T11:40:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]: dispatch 2024-03-24T11:40:34.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:33 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-03-24T11:40:34.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.262 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:34.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: pgmap v1002: 158 pgs: 158 active+clean; 127 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 943 op/s 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: osdmap e453: 8 total, 8 up, 6 in 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]: dispatch 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]: dispatch 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]: dispatch 2024-03-24T11:40:34.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]: dispatch 2024-03-24T11:40:34.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:33 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-03-24T11:40:34.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T11:40:34.363 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.391 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:34.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:34.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.519 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:34.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:34.586 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:34.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:34.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:40:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:40:34.645 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:34.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:34.771 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:34.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:34.898 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:34.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:35.000 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:35.065 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:35.130 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:40:35.130 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:40:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]': finished 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]': finished 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]': finished 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[29608]: osdmap e454: 8 total, 8 up, 6 in 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]': finished 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]': finished 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]': finished 2024-03-24T11:40:35.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[37907]: osdmap e454: 8 total, 8 up, 6 in 2024-03-24T11:40:35.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:35.132 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:34 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:35.150 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:40:35.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:35.169 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:40:35.189 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:40:35.193 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:35.210 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:40:35.228 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:40:35.248 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:40:35.268 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:40:35.269 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:40:35.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:35.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:34 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [5, 6]}]': finished 2024-03-24T11:40:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:34 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 2]}]': finished 2024-03-24T11:40:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:34 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]': finished 2024-03-24T11:40:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:34 smithi181 ceph-mon[33998]: osdmap e454: 8 total, 8 up, 6 in 2024-03-24T11:40:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:34 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:40:35.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:34 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:35.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.447 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:35.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:35.548 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:35.572 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:35.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.699 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:35.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:35.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.825 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:35.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:35.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T11:40:35.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:35.962 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:35.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:36.064 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:36.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:35 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:35 smithi181 ceph-mon[33998]: pgmap v1005: 158 pgs: 158 active+clean; 13 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-24T11:40:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:35 smithi181 ceph-mon[33998]: osdmap e455: 8 total, 8 up, 6 in 2024-03-24T11:40:36.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:35 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1625353452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:36.088 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:36.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:36.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:36.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[37907]: pgmap v1005: 158 pgs: 158 active+clean; 13 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-24T11:40:36.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[37907]: osdmap e455: 8 total, 8 up, 6 in 2024-03-24T11:40:36.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1625353452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[29608]: pgmap v1005: 158 pgs: 158 active+clean; 13 MiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-24T11:40:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[29608]: osdmap e455: 8 total, 8 up, 6 in 2024-03-24T11:40:36.210 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:35 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1625353452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:36.214 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:36.316 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:36.317 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:40:36.317 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:40:36.317 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-03-24T11:40:36.317 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-03-24T11:40:36.317 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:40:36.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:36.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.466 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:36.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:36.544 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-24T11:40:36.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-24T11:40:36.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:36.593 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:36.633 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:36.665 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:36.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:36.719 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:36.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:36.844 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:36.946 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:36.970 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:36.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:37.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T11:40:37.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.112 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:37.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:37.213 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:37.238 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:37.289 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:37.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:37.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:37.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:40:37.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:40:37.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[29608]: osdmap e456: 8 total, 8 up, 6 in 2024-03-24T11:40:37.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[29608]: pgmap v1008: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 507 op/s 2024-03-24T11:40:37.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:40:37.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:40:37.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[37907]: osdmap e456: 8 total, 8 up, 6 in 2024-03-24T11:40:37.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:37 smithi179 ceph-mon[37907]: pgmap v1008: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 507 op/s 2024-03-24T11:40:37.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:37.492 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:37.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:37.594 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:37.618 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:37 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:40:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:37 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:40:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:37 smithi181 ceph-mon[33998]: osdmap e456: 8 total, 8 up, 6 in 2024-03-24T11:40:37.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:37 smithi181 ceph-mon[33998]: pgmap v1008: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 507 op/s 2024-03-24T11:40:37.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.744 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:37.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:37.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:37.871 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:37.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:37.998 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:37.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:38.099 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:38.124 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:38.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:38.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:38.249 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:38.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:38.350 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:38.377 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:38.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:38.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:38.505 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:38.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:38.586 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T11:40:38.606 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:38.653 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:38.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:38.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:38.778 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:38.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:38.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:38.903 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:38.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:39.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:39.029 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:39.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:39.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:39.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:39.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:39.156 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:39.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:39.257 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:39.281 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:39.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:39.382 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:39.406 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:39.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:39.460 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:39.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:39.531 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:39.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:39.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:39.657 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:39.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:39.700 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:39 smithi179 ceph-mon[29608]: pgmap v1009: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 924 B/s rd, 0 B/s wr, 366 op/s 2024-03-24T11:40:39.700 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:39 smithi179 ceph-mon[37907]: pgmap v1009: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 924 B/s rd, 0 B/s wr, 366 op/s 2024-03-24T11:40:39.706 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:40:39.706 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213978 10.0 10:028c0284:::benchmark_data_smithi179_155357_object103482:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.291978+0000", 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 253.79924852299999, 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0690288450000001, 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.707 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213978 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291978+0000", 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291978+0000", 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291979+0000", 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.649e-06 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291985+0000", 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1100000000000002e-06 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.291999+0000", 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4358999999999999e-05 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292007+0000", 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4989999999999994e-06 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.531437+0000", 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23942971699999999 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.531447+0000", 2024-03-24T11:40:39.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0206e-05 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.360935+0000", 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82948748999999999 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.361007+0000", 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1815000000000003e-05 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.711 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213982 10.1d 10:bb1e392b:::benchmark_data_smithi179_155357_object103486:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.292022+0000", 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 253.79920393699999, 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1007629329999999, 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213982 2024-03-24T11:40:39.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292022+0000", 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292022+0000", 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292023+0000", 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9499999999999998e-07 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292025+0000", 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5730000000000002e-06 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292031+0000", 2024-03-24T11:40:39.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1099999999999999e-06 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.292036+0000", 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9849999999999997e-06 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530017+0000", 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23798054599999999 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530025+0000", 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8890000000000002e-06 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.392716+0000", 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86269092599999997 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.392785+0000", 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9509000000000001e-05 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213995 10.b 10:d077913b:::benchmark_data_smithi179_155357_object103499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.307895+0000", 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 253.783331379, 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.112235474, 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213995 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307895+0000", 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307895+0000", 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307897+0000", 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0269999999999998e-06 2024-03-24T11:40:39.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307908+0000", 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1311999999999999e-05 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307930+0000", 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1514000000000001e-05 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.307944+0000", 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3890000000000001e-05 2024-03-24T11:40:39.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.529700+0000", 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.221756113 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.529712+0000", 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2038000000000001e-05 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.365461+0000", 2024-03-24T11:40:39.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83574963800000002 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.365550+0000", 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8343000000000004e-05 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.365887+0000", 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033723100000000001 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.369902+0000", 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040151600000000003 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.413603+0000", 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043701146000000003 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420032+0000", 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064287880000000004 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420087+0000", 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4879000000000001e-05 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420130+0000", 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3395e-05 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213996 10.1a 10:5f2e433b:::benchmark_data_smithi179_155357_object103500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:39.723 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.315628+0000", 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 253.77559874100001, 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1033152850000001, 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213996 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.724 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315628+0000", 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315628+0000", 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315630+0000", 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9000000000000002e-06 2024-03-24T11:40:39.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315640+0000", 2024-03-24T11:40:39.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8430000000000005e-06 2024-03-24T11:40:39.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315664+0000", 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3883000000000002e-05 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.315678+0000", 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4174e-05 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.537200+0000", 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.22152160600000001 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.537210+0000", 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0222e-05 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399490+0000", 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86227972799999997 2024-03-24T11:40:39.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399621+0000", 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000130703 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399848+0000", 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022742899999999999 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.400929+0000", 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001080737 2024-03-24T11:40:39.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.401373+0000", 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000444719 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418875+0000", 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017501404000000002 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418909+0000", 2024-03-24T11:40:39.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4530999999999999e-05 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418943+0000", 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3405999999999998e-05 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:213998 10.1d 10:bc0250f7:::benchmark_data_smithi179_155357_object103502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.334402+0000", 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 253.75682402300001, 2024-03-24T11:40:39.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.084418715, 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 213998 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334402+0000", 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334402+0000", 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334405+0000", 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.385e-06 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334415+0000", 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0684e-05 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334434+0000", 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8297000000000001e-05 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.334445+0000", 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1430000000000001e-05 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530050+0000", 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.195604479 2024-03-24T11:40:39.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530056+0000", 2024-03-24T11:40:39.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4579999999999998e-06 2024-03-24T11:40:39.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.393146+0000", 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86308958800000002 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.393218+0000", 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2366000000000005e-05 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,4", 2024-03-24T11:40:39.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.393413+0000", 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000195145 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.394762+0000", 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013487939999999999 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.395209+0000", 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044712899999999999 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418726+0000", 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.023516927 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418778+0000", 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2154000000000002e-05 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.418821+0000", 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2879000000000001e-05 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.738 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214000 10.1b 10:d9e9d0b4:::benchmark_data_smithi179_155357_object103504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.344000+0000", 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 253.747226525, 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0764686059999999, 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.739 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214000 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344000+0000", 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344000+0000", 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344002+0000", 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6630000000000002e-06 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344013+0000", 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0387000000000001e-05 2024-03-24T11:40:39.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344050+0000", 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6708000000000002e-05 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.344070+0000", 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0489999999999999e-05 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530232+0000", 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18616228600000001 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.530240+0000", 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3300000000000001e-06 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.398610+0000", 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.86837029200000004 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.398766+0000", 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015581199999999999 2024-03-24T11:40:39.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,1", 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.399018+0000", 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000252031 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.400310+0000", 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012920869999999999 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.413623+0000", 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013313225999999999 2024-03-24T11:40:39.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420372+0000", 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0067491239999999996 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420426+0000", 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3949000000000001e-05 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.420468+0000", 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2221000000000001e-05 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:214001 10.a 10:54276112:::benchmark_data_smithi179_155357_object103505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:25.448752+0000", 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 253.642474628, 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96866780900000005, 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.747 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214001 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448752+0000", 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448752+0000", 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448754+0000", 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3470000000000001e-06 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448766+0000", 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1715999999999999e-05 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448798+0000", 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2740000000000002e-05 2024-03-24T11:40:39.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.448813+0000", 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4953000000000001e-05 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.536914+0000", 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.088100303000000005 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:25.536926+0000", 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2651e-05 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.367204+0000", 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83027731199999999 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.367339+0000", 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013548299999999999 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0,5", 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.367663+0000", 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032424599999999999 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.371158+0000", 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034944820000000001 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.400667+0000", 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.02950906 2024-03-24T11:40:39.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.417311+0000", 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016644042000000001 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.417374+0000", 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2748000000000003e-05 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:26.417419+0000", 2024-03-24T11:40:39.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5726000000000002e-05 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215434 10.1c 10.6089207c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 251.76616170599999, 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.998186252, 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.754 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215434 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325065+0000", 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.99e-07 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325069+0000", 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2710000000000001e-06 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325078+0000", 2024-03-24T11:40:39.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3209999999999994e-06 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325084+0000", 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6229999999999999e-06 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325311+0000", 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022635299999999999 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325319+0000", 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9119999999999998e-06 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323236+0000", 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99791718299999999 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323251+0000", 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.489e-05 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.758 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215438 10.1c 10.5cc036bc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.325397+0000", 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 251.765829303, 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99789192500000001, 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215438 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325397+0000", 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325397+0000", 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325398+0000", 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0999999999999998e-07 2024-03-24T11:40:39.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325402+0000", 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8249999999999998e-06 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325456+0000", 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4097000000000001e-05 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325506+0000", 2024-03-24T11:40:39.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0275999999999998e-05 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325551+0000", 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4663999999999998e-05 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.325557+0000", 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8950000000000001e-06 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323281+0000", 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99772327199999999 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323289+0000", 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1859999999999992e-06 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215480 10.1c 10.e4fdbf3c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.354574+0000", 2024-03-24T11:40:39.763 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 251.73665213000001, 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96874779600000005, 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215480 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354574+0000", 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354574+0000", 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354575+0000", 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1099999999999995e-07 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354578+0000", 2024-03-24T11:40:39.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3790000000000002e-06 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354585+0000", 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6420000000000001e-06 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354590+0000", 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4840000000000003e-06 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354632+0000", 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1233999999999999e-05 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.354637+0000", 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1109999999999999e-06 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323314+0000", 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96867741799999996 2024-03-24T11:40:39.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323322+0000", 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9170000000000006e-06 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215504 10.1c 10.f3f889bc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:27.370318+0000", 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 251.72090784400001, 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95304131700000005, 2024-03-24T11:40:39.768 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215504 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370318+0000", 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370318+0000", 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370319+0000", 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8599999999999998e-07 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370323+0000", 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9589999999999997e-06 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370332+0000", 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2799999999999992e-06 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370339+0000", 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7730000000000002e-06 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370386+0000", 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6835999999999997e-05 2024-03-24T11:40:39.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:27.370392+0000", 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5620000000000003e-06 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323352+0000", 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.95296023100000005 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.323360+0000", 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9899999999999997e-06 2024-03-24T11:40:39.772 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215439 10.1f 10:fa4c0bae:::benchmark_data_smithi179_155357_object104943:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318381+0000", 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 250.772845628, 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.125454446, 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.773 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215439 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318381+0000", 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318381+0000", 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318383+0000", 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-06 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318390+0000", 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318410+0000", 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9894e-05 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318423+0000", 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3715e-05 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318532+0000", 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108372 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-03-24T11:40:39.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318541+0000", 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6039999999999999e-06 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321920+0000", 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033789620000000001 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321930+0000", 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7510000000000003e-06 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434157+0000", 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1122265600000001 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for missing object", 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434437+0000", 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028081500000000001 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443751+0000", 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0093135800000000001 2024-03-24T11:40:39.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443835+0000", 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4095e-05 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215473 10.1f 10:f8f7c4ee:::benchmark_data_smithi179_155357_object104977:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 250.77277072699999, 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1260989669999999, 2024-03-24T11:40:39.779 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.780 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.780 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.780 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.781 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.781 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215473 2024-03-24T11:40:39.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318456+0000", 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3799999999999997e-07 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318460+0000", 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7479999999999999e-06 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318610+0000", 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000150109 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318621+0000", 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0552e-05 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318797+0000", 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000175892 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318802+0000", 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7950000000000004e-06 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321977+0000", 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003174718 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.321983+0000", 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4370000000000004e-06 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434809+0000", 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1128255279999999 2024-03-24T11:40:39.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.434952+0000", 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014316599999999999 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.435226+0000", 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000273773 2024-03-24T11:40:39.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439135+0000", 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039092069999999996 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443122+0000", 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039872429999999997 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443876+0000", 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000753132 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443980+0000", 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104318 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444253+0000", 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027285599999999998 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444360+0000", 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107435 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444555+0000", 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019442 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215489 10.1f 10:f9e29235:::benchmark_data_smithi179_155357_object104993:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318646+0000", 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 250.77257994300001, 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1283527250000001, 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215489 2024-03-24T11:40:39.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318646+0000", 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318646+0000", 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318647+0000", 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318652+0000", 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8199999999999996e-06 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318662+0000", 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9879999999999999e-06 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318669+0000", 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3379999999999999e-06 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318832+0000", 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000162351 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318837+0000", 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0080000000000002e-06 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322008+0000", 2024-03-24T11:40:39.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031713230000000002 2024-03-24T11:40:39.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322015+0000", 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6830000000000002e-06 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.436372+0000", 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1143569840000001 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.436486+0000", 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011397599999999999 2024-03-24T11:40:39.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.436692+0000", 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020638399999999999 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.443183+0000", 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064913000000000002 2024-03-24T11:40:39.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444114+0000", 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00093082500000000003 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444156+0000", 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1903999999999997e-05 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444629+0000", 2024-03-24T11:40:39.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047258800000000002 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446765+0000", 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021363089999999999 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446859+0000", 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4137999999999999e-05 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446999+0000", 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000139896 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.799 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215498 10.1f 10:faa9498f:::benchmark_data_smithi179_155357_object105002:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318681+0000", 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 250.772545331, 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1285948139999999, 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215498 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318681+0000", 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318681+0000", 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318682+0000", 2024-03-24T11:40:39.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2699999999999999e-07 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318685+0000", 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5200000000000002e-06 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318693+0000", 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6020000000000002e-06 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318699+0000", 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8089999999999998e-06 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318863+0000", 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016406799999999999 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318867+0000", 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8369999999999996e-06 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322038+0000", 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031707179999999999 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322044+0000", 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1639999999999998e-06 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.437844+0000", 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1157992670000001 2024-03-24T11:40:39.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.437969+0000", 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012585 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.438160+0000", 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000190197 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444029+0000", 2024-03-24T11:40:39.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058697849999999998 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444234+0000", 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020496199999999999 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444612+0000", 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037775500000000002 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444636+0000", 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3785e-05 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447057+0000", 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024206219999999999 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447129+0000", 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2862999999999997e-05 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447276+0000", 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014638300000000001 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.15621.0:215505 10.1f 10:fb4d5417:::benchmark_data_smithi179_155357_object105009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e373)", 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 250.77251930400001, 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1288068360000001, 2024-03-24T11:40:39.808 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.809 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.15621", 2024-03-24T11:40:39.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/3977226875", 2024-03-24T11:40:39.809 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215505 2024-03-24T11:40:39.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.809 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318707+0000", 2024-03-24T11:40:39.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2300000000000002e-07 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318710+0000", 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0680000000000001e-06 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318718+0000", 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5120000000000002e-06 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318730+0000", 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2084e-05 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318893+0000", 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016259799999999999 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.318897+0000", 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6e-06 2024-03-24T11:40:39.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322066+0000", 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031691530000000001 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:28.322073+0000", 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4529999999999999e-06 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439183+0000", 2024-03-24T11:40:39.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1171100140000001 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439307+0000", 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012411799999999999 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,2,3,7", 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.439496+0000", 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000189333 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444076+0000", 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004579301 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444641+0000", 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056571499999999995 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.444687+0000", 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5529999999999999e-05 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.446602+0000", 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001915017 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447324+0000", 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00072233799999999999 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447383+0000", 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9181999999999997e-05 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:36:29.447514+0000", 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013039700000000001 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25474.0:190390 11.3 11:c48347a1:::benchmark_data_smithi179_178435_object70092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-24T11:40:39.817 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:39:39.406176+0000", 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.685050576000002, 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0120307120000001, 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25474", 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/287749777", 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190390 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406176+0000", 2024-03-24T11:40:39.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406176+0000", 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406177+0000", 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8299999999999995e-07 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406181+0000", 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1710000000000004e-06 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406191+0000", 2024-03-24T11:40:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0319000000000001e-05 2024-03-24T11:40:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406199+0000", 2024-03-24T11:40:39.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6009999999999997e-06 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406252+0000", 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3418999999999998e-05 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.406261+0000", 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7739999999999993e-06 2024-03-24T11:40:39.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.409737+0000", 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.003475672 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.409877+0000", 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014044099999999999 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.410017+0000", 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013992 2024-03-24T11:40:39.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.414392+0000", 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043747079999999997 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.414438+0000", 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6428999999999999e-05 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418124+0000", 2024-03-24T11:40:39.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003685935 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418169+0000", 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5469000000000001e-05 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418206+0000", 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6971000000000002e-05 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25474.0:190422 11.3 11:c7481360:::benchmark_data_smithi179_178435_object70124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:39:39.437175+0000", 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.654050843, 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98127595499999998, 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25474", 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/287749777", 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190422 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437175+0000", 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437175+0000", 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437177+0000", 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.454e-06 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437181+0000", 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9330000000000003e-06 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437195+0000", 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4635e-05 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437204+0000", 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7520000000000002e-06 2024-03-24T11:40:39.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437270+0000", 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5665000000000002e-05 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.437279+0000", 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7369999999999995e-06 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.412433+0000", 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.97515446699999997 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.412554+0000", 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121347 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.412683+0000", 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012858499999999999 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418242+0000", 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055589009999999998 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418334+0000", 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1744999999999997e-05 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418394+0000", 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0137999999999999e-05 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418427+0000", 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3192999999999999e-05 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418451+0000", 2024-03-24T11:40:39.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4403e-05 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25474.0:190425 11.3 11:ca9c2142:::benchmark_data_smithi179_178435_object70127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:39:39.438521+0000", 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.652705427999997, 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98729784799999998, 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25474", 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/287749777", 2024-03-24T11:40:39.833 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190425 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438521+0000", 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438521+0000", 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438522+0000", 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8500000000000001e-07 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438525+0000", 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4000000000000001e-06 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438535+0000", 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0254999999999999e-05 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438541+0000", 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6200000000000004e-06 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438582+0000", 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0880999999999997e-05 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.438587+0000", 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3510000000000001e-06 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.414477+0000", 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.97588959500000005 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.414544+0000", 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7249999999999995e-05 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.414612+0000", 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7743999999999994e-05 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418381+0000", 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037690800000000002 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418457+0000", 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6663999999999996e-05 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.425707+0000", 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0072500079999999996 2024-03-24T11:40:39.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.425774+0000", 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6743999999999997e-05 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.425819+0000", 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4570999999999997e-05 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.25474.0:190436 11.3 11:c1e4c4fb:::benchmark_data_smithi179_178435_object70138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-03-24T11:39:39.448187+0000", 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.643038947000001, 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.98370070899999995, 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.25474", 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.179:0/287749777", 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190436 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448187+0000", 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448187+0000", 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448189+0000", 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4109999999999999e-06 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448193+0000", 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.391e-06 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448205+0000", 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2092e-05 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448213+0000", 2024-03-24T11:40:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0509999999999997e-06 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448266+0000", 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2716999999999997e-05 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:39.448272+0000", 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2670000000000003e-06 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.416059+0000", 2024-03-24T11:40:39.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.96778705700000001 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.416257+0000", 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019744599999999999 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 1,5", 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.416409+0000", 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015257199999999999 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-03-24T11:40:39.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.418461+0000", 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020513979999999999 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.425861+0000", 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0074001539999999999 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.431784+0000", 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059235360000000001 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-03-24T11:40:39.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.431844+0000", 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9191000000000003e-05 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-03-24T11:39:40.431888+0000", 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4425999999999998e-05 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-03-24T11:40:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:40:39.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:39.874 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:39.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:39 smithi181 ceph-mon[33998]: pgmap v1009: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 924 B/s rd, 0 B/s wr, 366 op/s 2024-03-24T11:40:39.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.003 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:40.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:40.104 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.130 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:40.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:40.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.255 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:40.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:40.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.382 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:40.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:40.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.509 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:40.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:40.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.639 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:40.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:40.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.764 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:40.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:40.865 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:40.889 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:40.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:40.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T11:40:40.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.016 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:41.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:41.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.145 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:41.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:41.157 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:41.246 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.271 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:41.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:41.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:41.397 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:41.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.524 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:41.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:41.565 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is set 2024-03-24T11:40:41.571 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T11:40:41.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:41.649 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:41.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.777 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:41.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:41.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[29608]: pgmap v1010: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 782 B/s rd, 0 B/s wr, 31 op/s 2024-03-24T11:40:41.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/996513847' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:41.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2507456940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:40:41.847 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:40:41.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[37907]: pgmap v1010: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 782 B/s rd, 0 B/s wr, 31 op/s 2024-03-24T11:40:41.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/996513847' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:41.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2507456940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:40:41.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:41 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:40:41.877 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:41 smithi181 ceph-mon[33998]: pgmap v1010: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 782 B/s rd, 0 B/s wr, 31 op/s 2024-03-24T11:40:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/996513847' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:40:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:41 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2507456940' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:40:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:41 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:40:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:41.904 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:41.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:42.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.030 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:42.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:42.132 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.158 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:42.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:42.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:42.284 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:42.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.411 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:42.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:42.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.538 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:42.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:42.639 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:42.662 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:42.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:42.789 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:42.791 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-03-24T11:40:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-03-24T11:40:42.791 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-03-24T11:40:42.792 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-03-24T11:40:42.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:42 smithi181 ceph-mon[33998]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:42.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:42 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/996513847' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:40:42.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:42 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:40:42.792 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:42 smithi181 ceph-mon[33998]: osdmap e457: 8 total, 8 up, 6 in 2024-03-24T11:40:42.804 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[29608]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:42.804 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/996513847' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:40:42.804 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:40:42.804 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[29608]: osdmap e457: 8 total, 8 up, 6 in 2024-03-24T11:40:42.804 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[37907]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:42.804 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/996513847' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T11:40:42.804 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:40:42.805 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:42 smithi179 ceph-mon[37907]: osdmap e457: 8 total, 8 up, 6 in 2024-03-24T11:40:42.891 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:42.917 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:42.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:43.018 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.045 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:43.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:43.146 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.171 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:43.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:43.272 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:43.297 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:43.399 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:43.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T11:40:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.465 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:43.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:43.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T11:40:43.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5 2024-03-24T11:40:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.592 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:43.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.5' 2024-03-24T11:40:43.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:43.699 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:43 smithi179 ceph-mon[29608]: pgmap v1012: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 680 B/s rd, 0 B/s wr, 27 op/s 2024-03-24T11:40:43.699 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:43 smithi179 ceph-mon[37907]: pgmap v1012: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 680 B/s rd, 0 B/s wr, 27 op/s 2024-03-24T11:40:43.706 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:43.722 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:43.745 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:43.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.849 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:43.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:43.867 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:43 smithi181 ceph-mon[33998]: pgmap v1012: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 680 B/s rd, 0 B/s wr, 27 op/s 2024-03-24T11:40:43.950 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:43.979 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:43.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:44.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.106 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:44.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:44.207 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.233 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:44.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:44.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:44.358 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:44.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:40:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:40:44.459 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.484 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:44.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:44.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:44.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.609 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:44.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:44.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:44.734 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:44.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:44 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:40:44.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:44 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:40:44.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:44 smithi181 ceph-mon[33998]: osdmap e458: 8 total, 8 up, 6 in 2024-03-24T11:40:44.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:44 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3894162613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:44.807 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:40:44.831 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:40:44.831 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:40:44.831 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[29608]: osdmap e458: 8 total, 8 up, 6 in 2024-03-24T11:40:44.831 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3894162613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:44.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:40:44.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:40:44.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[37907]: osdmap e458: 8 total, 8 up, 6 in 2024-03-24T11:40:44.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:44 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3894162613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T11:40:44.836 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.860 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:44.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:44.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:44.987 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:44.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:45.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:45.114 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:45.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:45.123 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:40:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:45.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:45.242 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:45.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:45.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:45.367 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:45.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:45.373 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:45.373 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:45.468 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:45.493 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:45.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:45.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.0: {} 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.1: {} 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.2: {} 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.3: {} 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.4: {} 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.5: {} 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.6: {} 2024-03-24T11:40:45.598 INFO:teuthology.orchestra.run.smithi179.stdout:osd.7: {} 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T11:40:45.599 INFO:teuthology.orchestra.run.smithi179.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T11:40:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:45.620 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:45.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:45 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:40:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:45 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:40:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:45 smithi181 ceph-mon[33998]: pgmap v1014: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:45.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:45 smithi181 ceph-mon[33998]: osdmap e459: 8 total, 8 up, 6 in 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[29608]: pgmap v1014: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[29608]: osdmap e459: 8 total, 8 up, 6 in 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:45.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[37907]: pgmap v1014: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:45.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:45 smithi179 ceph-mon[37907]: osdmap e459: 8 total, 8 up, 6 in 2024-03-24T11:40:45.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:45.748 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:45.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:45.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:45.875 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:45.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:45.976 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1 2024-03-24T11:40:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.001 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:46.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.1' 2024-03-24T11:40:46.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T11:40:46.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.127 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:46.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:46.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0 2024-03-24T11:40:46.252 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:46.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.0' 2024-03-24T11:40:46.295 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:40:46.354 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:46.379 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:46.462 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:46.464 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:46.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3 2024-03-24T11:40:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.507 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:46.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.3' 2024-03-24T11:40:46.604 INFO:teuthology.orchestra.run.smithi179.stderr:pool 'unique_pool_4' removed 2024-03-24T11:40:46.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2 2024-03-24T11:40:46.625 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:46 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3382728057' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:40:46.626 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:46 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:40:46.626 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:46 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3382728057' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:40:46.626 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:46 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:40:46.652 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: docker: command not found 2024-03-24T11:40:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi179 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.2' 2024-03-24T11:40:46.668 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:46 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3382728057' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:40:46.668 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:46 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:40:46.754 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.779 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:46.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:46.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6 2024-03-24T11:40:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:46.905 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:46.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.6' 2024-03-24T11:40:47.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:47.030 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:47.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:47.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7 2024-03-24T11:40:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:47.159 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:47.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.7' 2024-03-24T11:40:47.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4 2024-03-24T11:40:47.261 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-24T11:40:47.275 INFO:tasks.thrashosds:joining thrashosds 2024-03-24T11:40:47.299 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: docker: command not found 2024-03-24T11:40:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:40:47.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi181 with status 1: 'sudo docker kill -s 1 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd.4' 2024-03-24T11:40:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:47 smithi181 ceph-mon[33998]: pgmap v1016: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:47 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:40:47.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:47 smithi181 ceph-mon[33998]: osdmap e460: 8 total, 8 up, 6 in 2024-03-24T11:40:47.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:47 smithi179 ceph-mon[37907]: pgmap v1016: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:47 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:40:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:47 smithi179 ceph-mon[37907]: osdmap e460: 8 total, 8 up, 6 in 2024-03-24T11:40:47.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:47 smithi179 ceph-mon[29608]: pgmap v1016: 158 pgs: 158 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:40:47.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:47 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:40:47.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:47 smithi179 ceph-mon[29608]: osdmap e460: 8 total, 8 up, 6 in 2024-03-24T11:40:47.962 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-03-24T11:40:47.963 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T11:40:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:49 smithi181 ceph-mon[33998]: pgmap v1018: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:40:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:49 smithi179 ceph-mon[37907]: pgmap v1018: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:40:49.922 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:49.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:49 smithi179 ceph-mon[29608]: pgmap v1018: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T11:40:49.922 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:40:50.597 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T11:40:50.598 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:40:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2838266318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:50 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:50 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1848601493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:40:50.909 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2838266318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:50.909 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:50 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:50.909 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:50 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1848601493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:40:50.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2838266318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:50.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:50 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:50.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:50 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1848601493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:40:51.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T11:40:51.847 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[37907]: pgmap v1019: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[37907]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1848601493' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[37907]: osdmap e461: 8 total, 8 up, 6 in 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[29608]: pgmap v1019: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[29608]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1848601493' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:40:51.848 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:51 smithi179 ceph-mon[29608]: osdmap e461: 8 total, 8 up, 6 in 2024-03-24T11:40:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:51 smithi181 ceph-mon[33998]: pgmap v1019: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T11:40:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:51 smithi181 ceph-mon[33998]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T11:40:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:51 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1848601493' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:40:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:51 smithi181 ceph-mon[33998]: osdmap e461: 8 total, 8 up, 6 in 2024-03-24T11:40:51.917 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:52 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:40:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:52 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:40:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:52 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:40:52.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:52 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:40:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:52 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:40:52.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:52 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:40:53.659 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:53 smithi179 ceph-mon[29608]: pgmap v1021: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:40:53.660 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:53 smithi179 ceph-mon[37907]: pgmap v1021: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:40:53.873 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T11:40:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:53 smithi181 ceph-mon[33998]: pgmap v1021: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 516 B/s rd, 0 op/s 2024-03-24T11:40:54.131 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:54.602 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T11:40:54.603 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:40:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:40:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:54 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:40:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:54 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:40:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2888767242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:40:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:54 smithi181 ceph-mon[33998]: 5.9 scrub starts 2024-03-24T11:40:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:54 smithi181 ceph-mon[33998]: 5.9 scrub ok 2024-03-24T11:40:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2888767242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[29608]: 5.9 scrub starts 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[29608]: 5.9 scrub ok 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2888767242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[37907]: 5.9 scrub starts 2024-03-24T11:40:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:54 smithi179 ceph-mon[37907]: 5.9 scrub ok 2024-03-24T11:40:55.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:40:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:40:55.866 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 6.1f scrub ok 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: pgmap v1022: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2888767242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: osdmap e462: 8 total, 8 up, 6 in 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:40:55.867 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 6.1f scrub ok 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: pgmap v1022: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2888767242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: osdmap e462: 8 total, 8 up, 6 in 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:40:55.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:55 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 6.1f scrub ok 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: pgmap v1022: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2888767242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:40:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: osdmap e462: 8 total, 8 up, 6 in 2024-03-24T11:40:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:40:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:40:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:40:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:55 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:40:56.318 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T11:40:56.577 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:56.869 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:40:56.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:40:56.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[29608]: 4.1f scrub starts 2024-03-24T11:40:56.870 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[29608]: 4.1f scrub ok 2024-03-24T11:40:56.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:40:56.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:40:56.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[37907]: 4.1f scrub starts 2024-03-24T11:40:56.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:56 smithi179 ceph-mon[37907]: 4.1f scrub ok 2024-03-24T11:40:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:56 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:40:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:56 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:40:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:56 smithi181 ceph-mon[33998]: 4.1f scrub starts 2024-03-24T11:40:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:56 smithi181 ceph-mon[33998]: 4.1f scrub ok 2024-03-24T11:40:57.618 INFO:teuthology.orchestra.run.smithi179.stderr:noscrub is unset 2024-03-24T11:40:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:40:57.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:40:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: pgmap v1024: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:40:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:40:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1754466461' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: 6.3 scrub starts 2024-03-24T11:40:57.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[29608]: 6.3 scrub ok 2024-03-24T11:40:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:40:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:40:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: pgmap v1024: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:40:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:40:57.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1754466461' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:57.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: 6.3 scrub starts 2024-03-24T11:40:57.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:57 smithi179 ceph-mon[37907]: 6.3 scrub ok 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: pgmap v1024: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1754466461' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: 6.3 scrub starts 2024-03-24T11:40:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:57 smithi181 ceph-mon[33998]: 6.3 scrub ok 2024-03-24T11:40:58.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1754466461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: osdmap e463: 8 total, 8 up, 6 in 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:40:58.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:40:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:40:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1754466461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: osdmap e463: 8 total, 8 up, 6 in 2024-03-24T11:40:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:40:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:40:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:40:58.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:58 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1754466461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: osdmap e463: 8 total, 8 up, 6 in 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:40:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:58 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:40:59.143 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:40:59.634 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T11:40:59.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:59 smithi181 ceph-mon[33998]: pgmap v1026: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:40:59.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:59 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:40:59.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:59 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/849259879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:40:59.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:59 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:40:59.907 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:40:59 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:40:59.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[29608]: pgmap v1026: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/849259879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[37907]: pgmap v1026: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/849259879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:40:59.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:40:59 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:41:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:41:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:41:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:41:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: osdmap e464: 8 total, 8 up, 6 in 2024-03-24T11:41:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:41:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:41:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:41:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:41:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:41:00.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:00 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:41:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:41:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:41:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:41:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: osdmap e464: 8 total, 8 up, 6 in 2024-03-24T11:41:00.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: osdmap e464: 8 total, 8 up, 6 in 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:41:00.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:41:00.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:41:00.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:41:00.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:00 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:41:01.019 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T11:41:01.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:41:01.630 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: pgmap v1028: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:41:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:01 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: pgmap v1028: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:41:01.957 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: pgmap v1028: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:41:01.958 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:01 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:41:02.658 INFO:teuthology.orchestra.run.smithi179.stderr:nodeep-scrub is unset 2024-03-24T11:41:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:41:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4062651533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4062651533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:41:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:02 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:41:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:02 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:41:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:02 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:41:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:02 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4062651533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:41:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:02 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T11:41:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:02 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:41:02.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:02 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:41:03.805 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 4.17 scrub starts 2024-03-24T11:41:03.805 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 4.17 scrub ok 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: pgmap v1029: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: osdmap e465: 8 total, 8 up, 6 in 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 5.3 deep-scrub starts 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 5.3 deep-scrub ok 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:41:03.806 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 4.17 scrub starts 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 4.17 scrub ok 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: pgmap v1029: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: osdmap e465: 8 total, 8 up, 6 in 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 5.3 deep-scrub starts 2024-03-24T11:41:03.807 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 5.3 deep-scrub ok 2024-03-24T11:41:03.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:41:03.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:41:03.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:41:03.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:03 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:41:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 4.17 scrub starts 2024-03-24T11:41:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 4.17 scrub ok 2024-03-24T11:41:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: pgmap v1029: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 647 B/s rd, 0 op/s 2024-03-24T11:41:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T11:41:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:41:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: osdmap e465: 8 total, 8 up, 6 in 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 5.3 deep-scrub starts 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 5.3 deep-scrub ok 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:41:03.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:03 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:41:04.214 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:41:04.218 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:41:04.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:41:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:41:05.101 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":1030,"stamp":"2024-03-24T11:41:02.649950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2193,"num_read_kb":1945,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":661172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1512,"ondisk_log_size":1512,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4957360,"kb_used_data":7020,"kb_used_omap":9,"kb_used_meta":4950262,"kb_avail":557513552,"statfs":{"total":575970213888,"available":570893877248,"internally_reserved":0,"allocated":7188480,"data_stored":4822958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":5069068961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.148162"},"pg_stats":[{"pgid":"4.c","version":"421'30","reported_seq":565,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695923+0000","last_change":"2024-03-24T11:38:10.116849+0000","last_active":"2024-03-24T11:40:59.695923+0000","last_peered":"2024-03-24T11:40:59.695923+0000","last_clean":"2024-03-24T11:40:59.695923+0000","last_became_active":"2024-03-24T11:36:58.591623+0000","last_became_peered":"2024-03-24T11:36:58.591623+0000","last_unstale":"2024-03-24T11:40:59.695923+0000","last_undegraded":"2024-03-24T11:40:59.695923+0000","last_fullsized":"2024-03-24T11:40:59.695923+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:10.116699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.116699+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":869,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695758+0000","last_change":"2024-03-24T11:40:15.063982+0000","last_active":"2024-03-24T11:40:59.695758+0000","last_peered":"2024-03-24T11:40:59.695758+0000","last_clean":"2024-03-24T11:40:59.695758+0000","last_became_active":"2024-03-24T11:36:58.586602+0000","last_became_peered":"2024-03-24T11:36:58.586602+0000","last_unstale":"2024-03-24T11:40:59.695758+0000","last_undegraded":"2024-03-24T11:40:59.695758+0000","last_fullsized":"2024-03-24T11:40:59.695758+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:15.063776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.063776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:15.414894+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"429'10","reported_seq":1186,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.615520+0000","last_change":"2024-03-24T11:39:28.587290+0000","last_active":"2024-03-24T11:40:57.615520+0000","last_peered":"2024-03-24T11:40:57.615520+0000","last_clean":"2024-03-24T11:40:57.615520+0000","last_became_active":"2024-03-24T11:39:28.568543+0000","last_became_peered":"2024-03-24T11:39:28.568543+0000","last_unstale":"2024-03-24T11:40:57.615520+0000","last_undegraded":"2024-03-24T11:40:57.615520+0000","last_fullsized":"2024-03-24T11:40:57.615520+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'9","last_scrub_stamp":"2024-03-24T11:39:08.307817+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:39:08.307817+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":171,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791282+0000","last_change":"2024-03-24T11:38:01.792439+0000","last_active":"2024-03-24T11:40:57.791282+0000","last_peered":"2024-03-24T11:40:57.791282+0000","last_clean":"2024-03-24T11:40:57.791282+0000","last_became_active":"2024-03-24T11:36:58.590765+0000","last_became_peered":"2024-03-24T11:36:58.590765+0000","last_unstale":"2024-03-24T11:40:57.791282+0000","last_undegraded":"2024-03-24T11:40:57.791282+0000","last_fullsized":"2024-03-24T11:40:57.791282+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.792132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.792132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":948,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695151+0000","last_change":"2024-03-24T11:39:31.408536+0000","last_active":"2024-03-24T11:40:59.695151+0000","last_peered":"2024-03-24T11:40:59.695151+0000","last_clean":"2024-03-24T11:40:59.695151+0000","last_became_active":"2024-03-24T11:39:31.407869+0000","last_became_peered":"2024-03-24T11:39:31.407869+0000","last_unstale":"2024-03-24T11:40:59.695151+0000","last_undegraded":"2024-03-24T11:40:59.695151+0000","last_fullsized":"2024-03-24T11:40:59.695151+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:57.972898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:57.972898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"421'53","reported_seq":650,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.615362+0000","last_change":"2024-03-24T11:40:16.310832+0000","last_active":"2024-03-24T11:40:57.615362+0000","last_peered":"2024-03-24T11:40:57.615362+0000","last_clean":"2024-03-24T11:40:57.615362+0000","last_became_active":"2024-03-24T11:36:58.580651+0000","last_became_peered":"2024-03-24T11:36:58.580651+0000","last_unstale":"2024-03-24T11:40:57.615362+0000","last_undegraded":"2024-03-24T11:40:57.615362+0000","last_fullsized":"2024-03-24T11:40:57.615362+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'53","last_scrub_stamp":"2024-03-24T11:40:16.310682+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:40:16.310682+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.329404+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":920,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057747+0000","last_change":"2024-03-24T11:38:48.018756+0000","last_active":"2024-03-24T11:40:58.057747+0000","last_peered":"2024-03-24T11:40:58.057747+0000","last_clean":"2024-03-24T11:40:58.057747+0000","last_became_active":"2024-03-24T11:36:58.587097+0000","last_became_peered":"2024-03-24T11:36:58.587097+0000","last_unstale":"2024-03-24T11:40:58.057747+0000","last_undegraded":"2024-03-24T11:40:58.057747+0000","last_fullsized":"2024-03-24T11:40:58.057747+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:38:48.018617+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.018617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":892,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791291+0000","last_change":"2024-03-24T11:40:18.788766+0000","last_active":"2024-03-24T11:40:57.791291+0000","last_peered":"2024-03-24T11:40:57.791291+0000","last_clean":"2024-03-24T11:40:57.791291+0000","last_became_active":"2024-03-24T11:36:58.595572+0000","last_became_peered":"2024-03-24T11:36:58.595572+0000","last_unstale":"2024-03-24T11:40:57.791291+0000","last_undegraded":"2024-03-24T11:40:57.791291+0000","last_fullsized":"2024-03-24T11:40:57.791291+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:18.788516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:18.788516+0000","last_clean_scrub_stamp":"2024-03-24T11:40:18.788516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.145429+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":197,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696346+0000","last_change":"2024-03-24T11:38:52.569860+0000","last_active":"2024-03-24T11:40:59.696346+0000","last_peered":"2024-03-24T11:40:59.696346+0000","last_clean":"2024-03-24T11:40:59.696346+0000","last_became_active":"2024-03-24T11:36:58.582225+0000","last_became_peered":"2024-03-24T11:36:58.582225+0000","last_unstale":"2024-03-24T11:40:59.696346+0000","last_undegraded":"2024-03-24T11:40:59.696346+0000","last_fullsized":"2024-03-24T11:40:59.696346+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:52.569730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.569730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":1043,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792714+0000","last_change":"2024-03-24T11:38:03.034263+0000","last_active":"2024-03-24T11:40:57.792714+0000","last_peered":"2024-03-24T11:40:57.792714+0000","last_clean":"2024-03-24T11:40:57.792714+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:40:57.792714+0000","last_undegraded":"2024-03-24T11:40:57.792714+0000","last_fullsized":"2024-03-24T11:40:57.792714+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:38:03.034131+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:38:03.034131+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":577,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058238+0000","last_change":"2024-03-24T11:40:17.292448+0000","last_active":"2024-03-24T11:40:58.058238+0000","last_peered":"2024-03-24T11:40:58.058238+0000","last_clean":"2024-03-24T11:40:58.058238+0000","last_became_active":"2024-03-24T11:36:58.587116+0000","last_became_peered":"2024-03-24T11:36:58.587116+0000","last_unstale":"2024-03-24T11:40:58.058238+0000","last_undegraded":"2024-03-24T11:40:58.058238+0000","last_fullsized":"2024-03-24T11:40:58.058238+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:40:17.291968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.291968+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:43.826573+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":185,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.615215+0000","last_change":"2024-03-24T11:40:17.263847+0000","last_active":"2024-03-24T11:40:57.615215+0000","last_peered":"2024-03-24T11:40:57.615215+0000","last_clean":"2024-03-24T11:40:57.615215+0000","last_became_active":"2024-03-24T11:39:28.553193+0000","last_became_peered":"2024-03-24T11:39:28.553193+0000","last_unstale":"2024-03-24T11:40:57.615215+0000","last_undegraded":"2024-03-24T11:40:57.615215+0000","last_fullsized":"2024-03-24T11:40:57.615215+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:17.263730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.263730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:19.660517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":876,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.794104+0000","last_change":"2024-03-24T11:37:38.108397+0000","last_active":"2024-03-24T11:40:57.794104+0000","last_peered":"2024-03-24T11:40:57.794104+0000","last_clean":"2024-03-24T11:40:57.794104+0000","last_became_active":"2024-03-24T11:36:58.575297+0000","last_became_peered":"2024-03-24T11:36:58.575297+0000","last_unstale":"2024-03-24T11:40:57.794104+0000","last_undegraded":"2024-03-24T11:40:57.794104+0000","last_fullsized":"2024-03-24T11:40:57.794104+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:38.108325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:37:38.108325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":851,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695754+0000","last_change":"2024-03-24T11:40:17.160060+0000","last_active":"2024-03-24T11:40:59.695754+0000","last_peered":"2024-03-24T11:40:59.695754+0000","last_clean":"2024-03-24T11:40:59.695754+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:40:59.695754+0000","last_undegraded":"2024-03-24T11:40:59.695754+0000","last_fullsized":"2024-03-24T11:40:59.695754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:17.159943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.159943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:30.247287+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"421'46","reported_seq":1002,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791353+0000","last_change":"2024-03-24T11:40:20.669735+0000","last_active":"2024-03-24T11:40:57.791353+0000","last_peered":"2024-03-24T11:40:57.791353+0000","last_clean":"2024-03-24T11:40:57.791353+0000","last_became_active":"2024-03-24T11:39:28.585895+0000","last_became_peered":"2024-03-24T11:39:28.585895+0000","last_unstale":"2024-03-24T11:40:57.791353+0000","last_undegraded":"2024-03-24T11:40:57.791353+0000","last_fullsized":"2024-03-24T11:40:57.791353+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-24T11:40:20.669353+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:40:20.669353+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:22.302647+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":870,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695370+0000","last_change":"2024-03-24T11:39:28.589571+0000","last_active":"2024-03-24T11:40:59.695370+0000","last_peered":"2024-03-24T11:40:59.695370+0000","last_clean":"2024-03-24T11:40:59.695370+0000","last_became_active":"2024-03-24T11:39:28.587712+0000","last_became_peered":"2024-03-24T11:39:28.587712+0000","last_unstale":"2024-03-24T11:40:59.695370+0000","last_undegraded":"2024-03-24T11:40:59.695370+0000","last_fullsized":"2024-03-24T11:40:59.695370+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.005586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.005586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'9","reported_seq":256,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793716+0000","last_change":"2024-03-24T11:38:51.903763+0000","last_active":"2024-03-24T11:40:57.793716+0000","last_peered":"2024-03-24T11:40:57.793716+0000","last_clean":"2024-03-24T11:40:57.793716+0000","last_became_active":"2024-03-24T11:36:58.574227+0000","last_became_peered":"2024-03-24T11:36:58.574227+0000","last_unstale":"2024-03-24T11:40:57.793716+0000","last_undegraded":"2024-03-24T11:40:57.793716+0000","last_fullsized":"2024-03-24T11:40:57.793716+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-03-24T11:38:51.903639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:51.903639+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":866,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793397+0000","last_change":"2024-03-24T11:38:55.092421+0000","last_active":"2024-03-24T11:40:57.793397+0000","last_peered":"2024-03-24T11:40:57.793397+0000","last_clean":"2024-03-24T11:40:57.793397+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:40:57.793397+0000","last_undegraded":"2024-03-24T11:40:57.793397+0000","last_fullsized":"2024-03-24T11:40:57.793397+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:55.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:55.092309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":1034,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792828+0000","last_change":"2024-03-24T11:37:44.069186+0000","last_active":"2024-03-24T11:40:57.792828+0000","last_peered":"2024-03-24T11:40:57.792828+0000","last_clean":"2024-03-24T11:40:57.792828+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:40:57.792828+0000","last_undegraded":"2024-03-24T11:40:57.792828+0000","last_fullsized":"2024-03-24T11:40:57.792828+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:44.069073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.069073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":180,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695388+0000","last_change":"2024-03-24T11:39:28.567930+0000","last_active":"2024-03-24T11:40:59.695388+0000","last_peered":"2024-03-24T11:40:59.695388+0000","last_clean":"2024-03-24T11:40:59.695388+0000","last_became_active":"2024-03-24T11:39:28.567528+0000","last_became_peered":"2024-03-24T11:39:28.567528+0000","last_unstale":"2024-03-24T11:40:59.695388+0000","last_undegraded":"2024-03-24T11:40:59.695388+0000","last_fullsized":"2024-03-24T11:40:59.695388+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.082984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.082984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"421'42","reported_seq":945,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695515+0000","last_change":"2024-03-24T11:39:28.610844+0000","last_active":"2024-03-24T11:40:59.695515+0000","last_peered":"2024-03-24T11:40:59.695515+0000","last_clean":"2024-03-24T11:40:59.695515+0000","last_became_active":"2024-03-24T11:39:28.588159+0000","last_became_peered":"2024-03-24T11:39:28.588159+0000","last_unstale":"2024-03-24T11:40:59.695515+0000","last_undegraded":"2024-03-24T11:40:59.695515+0000","last_fullsized":"2024-03-24T11:40:59.695515+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:37:32.916579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:32.916579+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":870,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696164+0000","last_change":"2024-03-24T11:39:28.568155+0000","last_active":"2024-03-24T11:40:59.696164+0000","last_peered":"2024-03-24T11:40:59.696164+0000","last_clean":"2024-03-24T11:40:59.696164+0000","last_became_active":"2024-03-24T11:39:28.566304+0000","last_became_peered":"2024-03-24T11:39:28.566304+0000","last_unstale":"2024-03-24T11:40:59.696164+0000","last_undegraded":"2024-03-24T11:40:59.696164+0000","last_fullsized":"2024-03-24T11:40:59.696164+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:57.213332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.213332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":173,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695659+0000","last_change":"2024-03-24T11:40:13.056533+0000","last_active":"2024-03-24T11:40:59.695659+0000","last_peered":"2024-03-24T11:40:59.695659+0000","last_clean":"2024-03-24T11:40:59.695659+0000","last_became_active":"2024-03-24T11:36:58.574742+0000","last_became_peered":"2024-03-24T11:36:58.574742+0000","last_unstale":"2024-03-24T11:40:59.695659+0000","last_undegraded":"2024-03-24T11:40:59.695659+0000","last_fullsized":"2024-03-24T11:40:59.695659+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:13.056204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:40:13.056204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.868634+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":486,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.615612+0000","last_change":"2024-03-24T11:38:54.329326+0000","last_active":"2024-03-24T11:40:57.615612+0000","last_peered":"2024-03-24T11:40:57.615612+0000","last_clean":"2024-03-24T11:40:57.615612+0000","last_became_active":"2024-03-24T11:37:01.307977+0000","last_became_peered":"2024-03-24T11:37:01.307977+0000","last_unstale":"2024-03-24T11:40:57.615612+0000","last_undegraded":"2024-03-24T11:40:57.615612+0000","last_fullsized":"2024-03-24T11:40:57.615612+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:38:54.329204+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_clean_scrub_stamp":"2024-03-24T11:38:54.329204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"388'9","reported_seq":1132,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695462+0000","last_change":"2024-03-24T11:36:58.931284+0000","last_active":"2024-03-24T11:40:59.695462+0000","last_peered":"2024-03-24T11:40:59.695462+0000","last_clean":"2024-03-24T11:40:59.695462+0000","last_became_active":"2024-03-24T11:36:58.591546+0000","last_became_peered":"2024-03-24T11:36:58.591546+0000","last_unstale":"2024-03-24T11:40:59.695462+0000","last_undegraded":"2024-03-24T11:40:59.695462+0000","last_fullsized":"2024-03-24T11:40:59.695462+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:58.931185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:58.931185+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'42","reported_seq":593,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792892+0000","last_change":"2024-03-24T11:39:12.053807+0000","last_active":"2024-03-24T11:40:57.792892+0000","last_peered":"2024-03-24T11:40:57.792892+0000","last_clean":"2024-03-24T11:40:57.792892+0000","last_became_active":"2024-03-24T11:36:58.573987+0000","last_became_peered":"2024-03-24T11:36:58.573987+0000","last_unstale":"2024-03-24T11:40:57.792892+0000","last_undegraded":"2024-03-24T11:40:57.792892+0000","last_fullsized":"2024-03-24T11:40:57.792892+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'42","last_scrub_stamp":"2024-03-24T11:39:12.053617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:12.053617+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":916,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058276+0000","last_change":"2024-03-24T11:40:56.185901+0000","last_active":"2024-03-24T11:40:58.058276+0000","last_peered":"2024-03-24T11:40:58.058276+0000","last_clean":"2024-03-24T11:40:58.058276+0000","last_became_active":"2024-03-24T11:39:28.576255+0000","last_became_peered":"2024-03-24T11:39:28.576255+0000","last_unstale":"2024-03-24T11:40:58.058276+0000","last_undegraded":"2024-03-24T11:40:58.058276+0000","last_fullsized":"2024-03-24T11:40:58.058276+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:40:56.185811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:56.185811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:00.483005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":512,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058154+0000","last_change":"2024-03-24T11:40:57.351187+0000","last_active":"2024-03-24T11:40:58.058154+0000","last_peered":"2024-03-24T11:40:58.058154+0000","last_clean":"2024-03-24T11:40:58.058154+0000","last_became_active":"2024-03-24T11:39:28.572211+0000","last_became_peered":"2024-03-24T11:39:28.572211+0000","last_unstale":"2024-03-24T11:40:58.058154+0000","last_undegraded":"2024-03-24T11:40:58.058154+0000","last_fullsized":"2024-03-24T11:40:58.058154+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:57.351070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:57.351070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:14.159054+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1069,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695804+0000","last_change":"2024-03-24T11:39:31.390790+0000","last_active":"2024-03-24T11:40:59.695804+0000","last_peered":"2024-03-24T11:40:59.695804+0000","last_clean":"2024-03-24T11:40:59.695804+0000","last_became_active":"2024-03-24T11:39:31.390386+0000","last_became_peered":"2024-03-24T11:39:31.390386+0000","last_unstale":"2024-03-24T11:40:59.695804+0000","last_undegraded":"2024-03-24T11:40:59.695804+0000","last_fullsized":"2024-03-24T11:40:59.695804+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:42.045652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.045652+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":1153,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057282+0000","last_change":"2024-03-24T11:39:02.353239+0000","last_active":"2024-03-24T11:40:58.057282+0000","last_peered":"2024-03-24T11:40:58.057282+0000","last_clean":"2024-03-24T11:40:58.057282+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:40:58.057282+0000","last_undegraded":"2024-03-24T11:40:58.057282+0000","last_fullsized":"2024-03-24T11:40:58.057282+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:39:02.353110+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:32:42.033019+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.353110+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'30","reported_seq":954,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790789+0000","last_change":"2024-03-24T11:38:53.578501+0000","last_active":"2024-03-24T11:40:57.790789+0000","last_peered":"2024-03-24T11:40:57.790789+0000","last_clean":"2024-03-24T11:40:57.790789+0000","last_became_active":"2024-03-24T11:36:58.597641+0000","last_became_peered":"2024-03-24T11:36:58.597641+0000","last_unstale":"2024-03-24T11:40:57.790789+0000","last_undegraded":"2024-03-24T11:40:57.790789+0000","last_fullsized":"2024-03-24T11:40:57.790789+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:53.578365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.578365+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058205+0000","last_change":"2024-03-24T11:40:35.291321+0000","last_active":"2024-03-24T11:40:58.058205+0000","last_peered":"2024-03-24T11:40:58.058205+0000","last_clean":"2024-03-24T11:40:58.058205+0000","last_became_active":"2024-03-24T11:40:35.291074+0000","last_became_peered":"2024-03-24T11:40:35.291074+0000","last_unstale":"2024-03-24T11:40:58.058205+0000","last_undegraded":"2024-03-24T11:40:58.058205+0000","last_fullsized":"2024-03-24T11:40:58.058205+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:16.120029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_clean_scrub_stamp":"2024-03-24T11:39:16.120029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":876,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057405+0000","last_change":"2024-03-24T11:40:57.258985+0000","last_active":"2024-03-24T11:40:58.057405+0000","last_peered":"2024-03-24T11:40:58.057405+0000","last_clean":"2024-03-24T11:40:58.057405+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:40:58.057405+0000","last_undegraded":"2024-03-24T11:40:58.057405+0000","last_fullsized":"2024-03-24T11:40:58.057405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:40:57.258803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:57.258803+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:13.494335+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1094,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058496+0000","last_change":"2024-03-24T11:40:14.233673+0000","last_active":"2024-03-24T11:40:58.058496+0000","last_peered":"2024-03-24T11:40:58.058496+0000","last_clean":"2024-03-24T11:40:58.058496+0000","last_became_active":"2024-03-24T11:39:31.395565+0000","last_became_peered":"2024-03-24T11:39:31.395565+0000","last_unstale":"2024-03-24T11:40:58.058496+0000","last_undegraded":"2024-03-24T11:40:58.058496+0000","last_fullsized":"2024-03-24T11:40:58.058496+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:40:14.233591+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:40:14.233591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:29.225205+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":903,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058072+0000","last_change":"2024-03-24T11:40:21.311312+0000","last_active":"2024-03-24T11:40:58.058072+0000","last_peered":"2024-03-24T11:40:58.058072+0000","last_clean":"2024-03-24T11:40:58.058072+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:40:58.058072+0000","last_undegraded":"2024-03-24T11:40:58.058072+0000","last_fullsized":"2024-03-24T11:40:58.058072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:21.311216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:40:21.311216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:30.401945+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'39","reported_seq":973,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057825+0000","last_change":"2024-03-24T11:40:26.282042+0000","last_active":"2024-03-24T11:40:58.057825+0000","last_peered":"2024-03-24T11:40:58.057825+0000","last_clean":"2024-03-24T11:40:58.057825+0000","last_became_active":"2024-03-24T11:39:28.550731+0000","last_became_peered":"2024-03-24T11:39:28.550731+0000","last_unstale":"2024-03-24T11:40:58.057825+0000","last_undegraded":"2024-03-24T11:40:58.057825+0000","last_fullsized":"2024-03-24T11:40:58.057825+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:40:26.281902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:26.281902+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.052335+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":895,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791085+0000","last_change":"2024-03-24T11:37:49.656241+0000","last_active":"2024-03-24T11:40:57.791085+0000","last_peered":"2024-03-24T11:40:57.791085+0000","last_clean":"2024-03-24T11:40:57.791085+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:40:57.791085+0000","last_undegraded":"2024-03-24T11:40:57.791085+0000","last_fullsized":"2024-03-24T11:40:57.791085+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:49.656104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:49.656104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":875,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790702+0000","last_change":"2024-03-24T11:37:45.747922+0000","last_active":"2024-03-24T11:40:57.790702+0000","last_peered":"2024-03-24T11:40:57.790702+0000","last_clean":"2024-03-24T11:40:57.790702+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:40:57.790702+0000","last_undegraded":"2024-03-24T11:40:57.790702+0000","last_fullsized":"2024-03-24T11:40:57.790702+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:45.747799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:45.747799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1079,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058418+0000","last_change":"2024-03-24T11:39:14.270225+0000","last_active":"2024-03-24T11:40:58.058418+0000","last_peered":"2024-03-24T11:40:58.058418+0000","last_clean":"2024-03-24T11:40:58.058418+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:40:58.058418+0000","last_undegraded":"2024-03-24T11:40:58.058418+0000","last_fullsized":"2024-03-24T11:40:58.058418+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_clean_scrub_stamp":"2024-03-24T11:39:14.270084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":879,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.794123+0000","last_change":"2024-03-24T11:39:14.156792+0000","last_active":"2024-03-24T11:40:57.794123+0000","last_peered":"2024-03-24T11:40:57.794123+0000","last_clean":"2024-03-24T11:40:57.794123+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:40:57.794123+0000","last_undegraded":"2024-03-24T11:40:57.794123+0000","last_fullsized":"2024-03-24T11:40:57.794123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:14.156684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:39:14.156684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'36","reported_seq":446,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793614+0000","last_change":"2024-03-24T11:39:01.923217+0000","last_active":"2024-03-24T11:40:57.793614+0000","last_peered":"2024-03-24T11:40:57.793614+0000","last_clean":"2024-03-24T11:40:57.793614+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:40:57.793614+0000","last_undegraded":"2024-03-24T11:40:57.793614+0000","last_fullsized":"2024-03-24T11:40:57.793614+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'28","last_scrub_stamp":"2024-03-24T11:39:01.923083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.923083+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1173,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790504+0000","last_change":"2024-03-24T11:40:25.760465+0000","last_active":"2024-03-24T11:40:57.790504+0000","last_peered":"2024-03-24T11:40:57.790504+0000","last_clean":"2024-03-24T11:40:57.790504+0000","last_became_active":"2024-03-24T11:39:28.571981+0000","last_became_peered":"2024-03-24T11:39:28.571981+0000","last_unstale":"2024-03-24T11:40:57.790504+0000","last_undegraded":"2024-03-24T11:40:57.790504+0000","last_fullsized":"2024-03-24T11:40:57.790504+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:40:25.760218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:40:25.760218+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.257271+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":623,"num_read_kb":618,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":896,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057617+0000","last_change":"2024-03-24T11:39:06.423843+0000","last_active":"2024-03-24T11:40:58.057617+0000","last_peered":"2024-03-24T11:40:58.057617+0000","last_clean":"2024-03-24T11:40:58.057617+0000","last_became_active":"2024-03-24T11:36:58.572807+0000","last_became_peered":"2024-03-24T11:36:58.572807+0000","last_unstale":"2024-03-24T11:40:58.057617+0000","last_undegraded":"2024-03-24T11:40:58.057617+0000","last_fullsized":"2024-03-24T11:40:58.057617+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:06.423513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:39:06.423513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":883,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790974+0000","last_change":"2024-03-24T11:38:05.656269+0000","last_active":"2024-03-24T11:40:57.790974+0000","last_peered":"2024-03-24T11:40:57.790974+0000","last_clean":"2024-03-24T11:40:57.790974+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:40:57.790974+0000","last_undegraded":"2024-03-24T11:40:57.790974+0000","last_fullsized":"2024-03-24T11:40:57.790974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.656146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.656146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1090,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790895+0000","last_change":"2024-03-24T11:40:15.791385+0000","last_active":"2024-03-24T11:40:57.790895+0000","last_peered":"2024-03-24T11:40:57.790895+0000","last_clean":"2024-03-24T11:40:57.790895+0000","last_became_active":"2024-03-24T11:37:01.307925+0000","last_became_peered":"2024-03-24T11:37:01.307925+0000","last_unstale":"2024-03-24T11:40:57.790895+0000","last_undegraded":"2024-03-24T11:40:57.790895+0000","last_fullsized":"2024-03-24T11:40:57.790895+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:40:15.790434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.790434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:15.988035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":512,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.617299+0000","last_change":"2024-03-24T11:40:11.295320+0000","last_active":"2024-03-24T11:40:57.617299+0000","last_peered":"2024-03-24T11:40:57.617299+0000","last_clean":"2024-03-24T11:40:57.617299+0000","last_became_active":"2024-03-24T11:39:28.569858+0000","last_became_peered":"2024-03-24T11:39:28.569858+0000","last_unstale":"2024-03-24T11:40:57.617299+0000","last_undegraded":"2024-03-24T11:40:57.617299+0000","last_fullsized":"2024-03-24T11:40:57.617299+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:11.295201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.295201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:41.218881+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"429'50","reported_seq":1286,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058468+0000","last_change":"2024-03-24T11:40:55.208483+0000","last_active":"2024-03-24T11:40:58.058468+0000","last_peered":"2024-03-24T11:40:58.058468+0000","last_clean":"2024-03-24T11:40:58.058468+0000","last_became_active":"2024-03-24T11:39:28.564381+0000","last_became_peered":"2024-03-24T11:39:28.564381+0000","last_unstale":"2024-03-24T11:40:58.058468+0000","last_undegraded":"2024-03-24T11:40:58.058468+0000","last_fullsized":"2024-03-24T11:40:58.058468+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'50","last_scrub_stamp":"2024-03-24T11:40:55.208344+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:40:55.208344+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:18.277885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":876,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695333+0000","last_change":"2024-03-24T11:39:28.585112+0000","last_active":"2024-03-24T11:40:59.695333+0000","last_peered":"2024-03-24T11:40:59.695333+0000","last_clean":"2024-03-24T11:40:59.695333+0000","last_became_active":"2024-03-24T11:39:28.584724+0000","last_became_peered":"2024-03-24T11:39:28.584724+0000","last_unstale":"2024-03-24T11:40:59.695333+0000","last_undegraded":"2024-03-24T11:40:59.695333+0000","last_fullsized":"2024-03-24T11:40:59.695333+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.978638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.978638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":842,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057449+0000","last_change":"2024-03-24T11:39:31.397589+0000","last_active":"2024-03-24T11:40:58.057449+0000","last_peered":"2024-03-24T11:40:58.057449+0000","last_clean":"2024-03-24T11:40:58.057449+0000","last_became_active":"2024-03-24T11:39:31.397237+0000","last_became_peered":"2024-03-24T11:39:31.397237+0000","last_unstale":"2024-03-24T11:40:58.057449+0000","last_undegraded":"2024-03-24T11:40:58.057449+0000","last_fullsized":"2024-03-24T11:40:58.057449+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:39:03.394244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:39:03.394244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":889,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793935+0000","last_change":"2024-03-24T11:38:02.291361+0000","last_active":"2024-03-24T11:40:57.793935+0000","last_peered":"2024-03-24T11:40:57.793935+0000","last_clean":"2024-03-24T11:40:57.793935+0000","last_became_active":"2024-03-24T11:36:58.586913+0000","last_became_peered":"2024-03-24T11:36:58.586913+0000","last_unstale":"2024-03-24T11:40:57.793935+0000","last_undegraded":"2024-03-24T11:40:57.793935+0000","last_fullsized":"2024-03-24T11:40:57.793935+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:02.291240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:38:02.291240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":909,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.616123+0000","last_change":"2024-03-24T11:39:28.575114+0000","last_active":"2024-03-24T11:40:57.616123+0000","last_peered":"2024-03-24T11:40:57.616123+0000","last_clean":"2024-03-24T11:40:57.616123+0000","last_became_active":"2024-03-24T11:39:28.574632+0000","last_became_peered":"2024-03-24T11:39:28.574632+0000","last_unstale":"2024-03-24T11:40:57.616123+0000","last_undegraded":"2024-03-24T11:40:57.616123+0000","last_fullsized":"2024-03-24T11:40:57.616123+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:13.234123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:13.234123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"449'26","reported_seq":926,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.617015+0000","last_change":"2024-03-24T11:40:22.408424+0000","last_active":"2024-03-24T11:40:57.617015+0000","last_peered":"2024-03-24T11:40:57.617015+0000","last_clean":"2024-03-24T11:40:57.617015+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:40:57.617015+0000","last_undegraded":"2024-03-24T11:40:57.617015+0000","last_fullsized":"2024-03-24T11:40:57.617015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'26","last_scrub_stamp":"2024-03-24T11:40:22.408163+0000","last_deep_scrub":"449'26","last_deep_scrub_stamp":"2024-03-24T11:40:22.408163+0000","last_clean_scrub_stamp":"2024-03-24T11:40:22.408163+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:42.483835+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":920,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.615312+0000","last_change":"2024-03-24T11:40:15.313269+0000","last_active":"2024-03-24T11:40:57.615312+0000","last_peered":"2024-03-24T11:40:57.615312+0000","last_clean":"2024-03-24T11:40:57.615312+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:40:57.615312+0000","last_undegraded":"2024-03-24T11:40:57.615312+0000","last_fullsized":"2024-03-24T11:40:57.615312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:15.313154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.313154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:20.321870+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":895,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790530+0000","last_change":"2024-03-24T11:37:53.683206+0000","last_active":"2024-03-24T11:40:57.790530+0000","last_peered":"2024-03-24T11:40:57.790530+0000","last_clean":"2024-03-24T11:40:57.790530+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:40:57.790530+0000","last_undegraded":"2024-03-24T11:40:57.790530+0000","last_fullsized":"2024-03-24T11:40:57.790530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:53.683090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:53.683090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":504,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058476+0000","last_change":"2024-03-24T11:39:28.562110+0000","last_active":"2024-03-24T11:40:58.058476+0000","last_peered":"2024-03-24T11:40:58.058476+0000","last_clean":"2024-03-24T11:40:58.058476+0000","last_became_active":"2024-03-24T11:39:28.561579+0000","last_became_peered":"2024-03-24T11:39:28.561579+0000","last_unstale":"2024-03-24T11:40:58.058476+0000","last_undegraded":"2024-03-24T11:40:58.058476+0000","last_fullsized":"2024-03-24T11:40:58.058476+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:55.094992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:38:55.094992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":866,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057915+0000","last_change":"2024-03-24T11:40:13.361900+0000","last_active":"2024-03-24T11:40:58.057915+0000","last_peered":"2024-03-24T11:40:58.057915+0000","last_clean":"2024-03-24T11:40:58.057915+0000","last_became_active":"2024-03-24T11:39:30.375769+0000","last_became_peered":"2024-03-24T11:39:30.375769+0000","last_unstale":"2024-03-24T11:40:58.057915+0000","last_undegraded":"2024-03-24T11:40:58.057915+0000","last_fullsized":"2024-03-24T11:40:58.057915+0000","mapping_epoch":430,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:40:13.361480+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:40:13.361480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:25.011763+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":885,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058235+0000","last_change":"2024-03-24T11:40:18.336459+0000","last_active":"2024-03-24T11:40:58.058235+0000","last_peered":"2024-03-24T11:40:58.058235+0000","last_clean":"2024-03-24T11:40:58.058235+0000","last_became_active":"2024-03-24T11:36:58.594154+0000","last_became_peered":"2024-03-24T11:36:58.594154+0000","last_unstale":"2024-03-24T11:40:58.058235+0000","last_undegraded":"2024-03-24T11:40:58.058235+0000","last_fullsized":"2024-03-24T11:40:58.058235+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:18.336351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:18.336351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:26.769793+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"421'53","reported_seq":1260,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791187+0000","last_change":"2024-03-24T11:37:44.707841+0000","last_active":"2024-03-24T11:40:57.791187+0000","last_peered":"2024-03-24T11:40:57.791187+0000","last_clean":"2024-03-24T11:40:57.791187+0000","last_became_active":"2024-03-24T11:37:33.888447+0000","last_became_peered":"2024-03-24T11:37:33.888447+0000","last_unstale":"2024-03-24T11:40:57.791187+0000","last_undegraded":"2024-03-24T11:40:57.791187+0000","last_fullsized":"2024-03-24T11:40:57.791187+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'41","last_scrub_stamp":"2024-03-24T11:37:44.707693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:44.707693+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":179,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057818+0000","last_change":"2024-03-24T11:40:22.347875+0000","last_active":"2024-03-24T11:40:58.057818+0000","last_peered":"2024-03-24T11:40:58.057818+0000","last_clean":"2024-03-24T11:40:58.057818+0000","last_became_active":"2024-03-24T11:36:58.591467+0000","last_became_peered":"2024-03-24T11:36:58.591467+0000","last_unstale":"2024-03-24T11:40:58.057818+0000","last_undegraded":"2024-03-24T11:40:58.057818+0000","last_fullsized":"2024-03-24T11:40:58.057818+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:22.347618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_clean_scrub_stamp":"2024-03-24T11:40:22.347618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:43.963473+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":898,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793251+0000","last_change":"2024-03-24T11:39:05.008974+0000","last_active":"2024-03-24T11:40:57.793251+0000","last_peered":"2024-03-24T11:40:57.793251+0000","last_clean":"2024-03-24T11:40:57.793251+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:40:57.793251+0000","last_undegraded":"2024-03-24T11:40:57.793251+0000","last_fullsized":"2024-03-24T11:40:57.793251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:39:05.008874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:39:05.008874+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":864,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696136+0000","last_change":"2024-03-24T11:39:28.580748+0000","last_active":"2024-03-24T11:40:59.696136+0000","last_peered":"2024-03-24T11:40:59.696136+0000","last_clean":"2024-03-24T11:40:59.696136+0000","last_became_active":"2024-03-24T11:39:28.576107+0000","last_became_peered":"2024-03-24T11:39:28.576107+0000","last_unstale":"2024-03-24T11:40:59.696136+0000","last_undegraded":"2024-03-24T11:40:59.696136+0000","last_fullsized":"2024-03-24T11:40:59.696136+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:02.217239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.217239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1065,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792857+0000","last_change":"2024-03-24T11:37:55.105164+0000","last_active":"2024-03-24T11:40:57.792857+0000","last_peered":"2024-03-24T11:40:57.792857+0000","last_clean":"2024-03-24T11:40:57.792857+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:40:57.792857+0000","last_undegraded":"2024-03-24T11:40:57.792857+0000","last_fullsized":"2024-03-24T11:40:57.792857+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:55.105007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.105007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":932,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058050+0000","last_change":"2024-03-24T11:39:12.419766+0000","last_active":"2024-03-24T11:40:58.058050+0000","last_peered":"2024-03-24T11:40:58.058050+0000","last_clean":"2024-03-24T11:40:58.058050+0000","last_became_active":"2024-03-24T11:36:58.590879+0000","last_became_peered":"2024-03-24T11:36:58.590879+0000","last_unstale":"2024-03-24T11:40:58.058050+0000","last_undegraded":"2024-03-24T11:40:58.058050+0000","last_fullsized":"2024-03-24T11:40:58.058050+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:39:12.418838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:12.418838+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":892,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793432+0000","last_change":"2024-03-24T11:38:05.022910+0000","last_active":"2024-03-24T11:40:57.793432+0000","last_peered":"2024-03-24T11:40:57.793432+0000","last_clean":"2024-03-24T11:40:57.793432+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:40:57.793432+0000","last_undegraded":"2024-03-24T11:40:57.793432+0000","last_fullsized":"2024-03-24T11:40:57.793432+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:05.022785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.022785+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":175,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695373+0000","last_change":"2024-03-24T11:39:01.186035+0000","last_active":"2024-03-24T11:40:59.695373+0000","last_peered":"2024-03-24T11:40:59.695373+0000","last_clean":"2024-03-24T11:40:59.695373+0000","last_became_active":"2024-03-24T11:36:58.576200+0000","last_became_peered":"2024-03-24T11:36:58.576200+0000","last_unstale":"2024-03-24T11:40:59.695373+0000","last_undegraded":"2024-03-24T11:40:59.695373+0000","last_fullsized":"2024-03-24T11:40:59.695373+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:01.185721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.185721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":881,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057827+0000","last_change":"2024-03-24T11:40:16.413527+0000","last_active":"2024-03-24T11:40:58.057827+0000","last_peered":"2024-03-24T11:40:58.057827+0000","last_clean":"2024-03-24T11:40:58.057827+0000","last_became_active":"2024-03-24T11:39:28.567784+0000","last_became_peered":"2024-03-24T11:39:28.567784+0000","last_unstale":"2024-03-24T11:40:58.057827+0000","last_undegraded":"2024-03-24T11:40:58.057827+0000","last_fullsized":"2024-03-24T11:40:58.057827+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:16.413401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:16.413401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:41.494869+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":1013,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058131+0000","last_change":"2024-03-24T11:40:11.195573+0000","last_active":"2024-03-24T11:40:58.058131+0000","last_peered":"2024-03-24T11:40:58.058131+0000","last_clean":"2024-03-24T11:40:58.058131+0000","last_became_active":"2024-03-24T11:39:28.547142+0000","last_became_peered":"2024-03-24T11:39:28.547142+0000","last_unstale":"2024-03-24T11:40:58.058131+0000","last_undegraded":"2024-03-24T11:40:58.058131+0000","last_fullsized":"2024-03-24T11:40:58.058131+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'60","last_scrub_stamp":"2024-03-24T11:40:11.195421+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.195421+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:17.973047+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":502,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793764+0000","last_change":"2024-03-24T11:40:34.982216+0000","last_active":"2024-03-24T11:40:57.793764+0000","last_peered":"2024-03-24T11:40:57.793764+0000","last_clean":"2024-03-24T11:40:57.793764+0000","last_became_active":"2024-03-24T11:40:34.981873+0000","last_became_peered":"2024-03-24T11:40:34.981873+0000","last_unstale":"2024-03-24T11:40:57.793764+0000","last_undegraded":"2024-03-24T11:40:57.793764+0000","last_fullsized":"2024-03-24T11:40:57.793764+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:46.876805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:46.876805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":910,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.239005+0000","last_change":"2024-03-24T11:40:58.239024+0000","last_active":"2024-03-24T11:40:58.239005+0000","last_peered":"2024-03-24T11:40:58.239005+0000","last_clean":"2024-03-24T11:40:58.239005+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:40:58.239005+0000","last_undegraded":"2024-03-24T11:40:58.239005+0000","last_fullsized":"2024-03-24T11:40:58.239005+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:58.238935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:40:58.238935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:26.797965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":198,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695140+0000","last_change":"2024-03-24T11:39:28.587339+0000","last_active":"2024-03-24T11:40:59.695140+0000","last_peered":"2024-03-24T11:40:59.695140+0000","last_clean":"2024-03-24T11:40:59.695140+0000","last_became_active":"2024-03-24T11:39:28.575619+0000","last_became_peered":"2024-03-24T11:39:28.575619+0000","last_unstale":"2024-03-24T11:40:59.695140+0000","last_undegraded":"2024-03-24T11:40:59.695140+0000","last_fullsized":"2024-03-24T11:40:59.695140+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.123323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":197,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058166+0000","last_change":"2024-03-24T11:40:53.244059+0000","last_active":"2024-03-24T11:40:58.058166+0000","last_peered":"2024-03-24T11:40:58.058166+0000","last_clean":"2024-03-24T11:40:58.058166+0000","last_became_active":"2024-03-24T11:36:42.470376+0000","last_became_peered":"2024-03-24T11:36:42.470376+0000","last_unstale":"2024-03-24T11:40:58.058166+0000","last_undegraded":"2024-03-24T11:40:58.058166+0000","last_fullsized":"2024-03-24T11:40:58.058166+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:53.243956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:53.243956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:03.661987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"421'24","reported_seq":567,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790424+0000","last_change":"2024-03-24T11:38:52.571057+0000","last_active":"2024-03-24T11:40:57.790424+0000","last_peered":"2024-03-24T11:40:57.790424+0000","last_clean":"2024-03-24T11:40:57.790424+0000","last_became_active":"2024-03-24T11:36:58.593181+0000","last_became_peered":"2024-03-24T11:36:58.593181+0000","last_unstale":"2024-03-24T11:40:57.790424+0000","last_undegraded":"2024-03-24T11:40:57.790424+0000","last_fullsized":"2024-03-24T11:40:57.790424+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:38:52.570919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.570919+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":179,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791565+0000","last_change":"2024-03-24T11:39:28.579352+0000","last_active":"2024-03-24T11:40:57.791565+0000","last_peered":"2024-03-24T11:40:57.791565+0000","last_clean":"2024-03-24T11:40:57.791565+0000","last_became_active":"2024-03-24T11:39:28.578963+0000","last_became_peered":"2024-03-24T11:39:28.578963+0000","last_unstale":"2024-03-24T11:40:57.791565+0000","last_undegraded":"2024-03-24T11:40:57.791565+0000","last_fullsized":"2024-03-24T11:40:57.791565+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:47.673879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:37:47.673879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":857,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695975+0000","last_change":"2024-03-24T11:40:26.396360+0000","last_active":"2024-03-24T11:40:59.695975+0000","last_peered":"2024-03-24T11:40:59.695975+0000","last_clean":"2024-03-24T11:40:59.695975+0000","last_became_active":"2024-03-24T11:39:28.581732+0000","last_became_peered":"2024-03-24T11:39:28.581732+0000","last_unstale":"2024-03-24T11:40:59.695975+0000","last_undegraded":"2024-03-24T11:40:59.695975+0000","last_fullsized":"2024-03-24T11:40:59.695975+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:26.396256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:26.396256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:39.252686+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"429'12","reported_seq":677,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057597+0000","last_change":"2024-03-24T11:40:54.205084+0000","last_active":"2024-03-24T11:40:58.057597+0000","last_peered":"2024-03-24T11:40:58.057597+0000","last_clean":"2024-03-24T11:40:58.057597+0000","last_became_active":"2024-03-24T11:39:28.577500+0000","last_became_peered":"2024-03-24T11:39:28.577500+0000","last_unstale":"2024-03-24T11:40:58.057597+0000","last_undegraded":"2024-03-24T11:40:58.057597+0000","last_fullsized":"2024-03-24T11:40:58.057597+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'12","last_scrub_stamp":"2024-03-24T11:40:54.204976+0000","last_deep_scrub":"357'10","last_deep_scrub_stamp":"2024-03-24T11:35:53.386928+0000","last_clean_scrub_stamp":"2024-03-24T11:40:54.204976+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:06.153466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"421'46","reported_seq":973,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792584+0000","last_change":"2024-03-24T11:39:28.598972+0000","last_active":"2024-03-24T11:40:57.792584+0000","last_peered":"2024-03-24T11:40:57.792584+0000","last_clean":"2024-03-24T11:40:57.792584+0000","last_became_active":"2024-03-24T11:39:28.563427+0000","last_became_peered":"2024-03-24T11:39:28.563427+0000","last_unstale":"2024-03-24T11:40:57.792584+0000","last_undegraded":"2024-03-24T11:40:57.792584+0000","last_fullsized":"2024-03-24T11:40:57.792584+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'34","last_scrub_stamp":"2024-03-24T11:38:00.140160+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:38:00.140160+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":399,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058325+0000","last_change":"2024-03-24T11:40:21.348781+0000","last_active":"2024-03-24T11:40:58.058325+0000","last_peered":"2024-03-24T11:40:58.058325+0000","last_clean":"2024-03-24T11:40:58.058325+0000","last_became_active":"2024-03-24T11:39:28.582617+0000","last_became_peered":"2024-03-24T11:39:28.582617+0000","last_unstale":"2024-03-24T11:40:58.058325+0000","last_undegraded":"2024-03-24T11:40:58.058325+0000","last_fullsized":"2024-03-24T11:40:58.058325+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:40:21.348646+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:40:21.348646+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:35.106715+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":875,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793453+0000","last_change":"2024-03-24T11:38:04.034506+0000","last_active":"2024-03-24T11:40:57.793453+0000","last_peered":"2024-03-24T11:40:57.793453+0000","last_clean":"2024-03-24T11:40:57.793453+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:40:57.793453+0000","last_undegraded":"2024-03-24T11:40:57.793453+0000","last_fullsized":"2024-03-24T11:40:57.793453+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:04.034411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.034411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":895,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696341+0000","last_change":"2024-03-24T11:39:28.607055+0000","last_active":"2024-03-24T11:40:59.696341+0000","last_peered":"2024-03-24T11:40:59.696341+0000","last_clean":"2024-03-24T11:40:59.696341+0000","last_became_active":"2024-03-24T11:39:28.583847+0000","last_became_peered":"2024-03-24T11:39:28.583847+0000","last_unstale":"2024-03-24T11:40:59.696341+0000","last_undegraded":"2024-03-24T11:40:59.696341+0000","last_fullsized":"2024-03-24T11:40:59.696341+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:35:53.797708+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:35:53.797708+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":910,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057296+0000","last_change":"2024-03-24T11:40:26.287970+0000","last_active":"2024-03-24T11:40:58.057296+0000","last_peered":"2024-03-24T11:40:58.057296+0000","last_clean":"2024-03-24T11:40:58.057296+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:40:58.057296+0000","last_undegraded":"2024-03-24T11:40:58.057296+0000","last_fullsized":"2024-03-24T11:40:58.057296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:26.287872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:26.287872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:41.149626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":878,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792409+0000","last_change":"2024-03-24T11:40:22.248773+0000","last_active":"2024-03-24T11:40:57.792409+0000","last_peered":"2024-03-24T11:40:57.792409+0000","last_clean":"2024-03-24T11:40:57.792409+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:40:57.792409+0000","last_undegraded":"2024-03-24T11:40:57.792409+0000","last_fullsized":"2024-03-24T11:40:57.792409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:22.248686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:22.248686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:45.429437+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":518,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058583+0000","last_change":"2024-03-24T11:40:24.442702+0000","last_active":"2024-03-24T11:40:58.058583+0000","last_peered":"2024-03-24T11:40:58.058583+0000","last_clean":"2024-03-24T11:40:58.058583+0000","last_became_active":"2024-03-24T11:39:28.547484+0000","last_became_peered":"2024-03-24T11:39:28.547484+0000","last_unstale":"2024-03-24T11:40:58.058583+0000","last_undegraded":"2024-03-24T11:40:58.058583+0000","last_fullsized":"2024-03-24T11:40:58.058583+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:24.442599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:24.442599+0000","last_clean_scrub_stamp":"2024-03-24T11:40:24.442599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:45.397849+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"421'25","reported_seq":936,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058413+0000","last_change":"2024-03-24T11:39:28.590958+0000","last_active":"2024-03-24T11:40:58.058413+0000","last_peered":"2024-03-24T11:40:58.058413+0000","last_clean":"2024-03-24T11:40:58.058413+0000","last_became_active":"2024-03-24T11:39:28.564216+0000","last_became_peered":"2024-03-24T11:39:28.564216+0000","last_unstale":"2024-03-24T11:40:58.058413+0000","last_undegraded":"2024-03-24T11:40:58.058413+0000","last_fullsized":"2024-03-24T11:40:58.058413+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:38:57.280974+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.280974+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":187,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057374+0000","last_change":"2024-03-24T11:40:12.210986+0000","last_active":"2024-03-24T11:40:58.057374+0000","last_peered":"2024-03-24T11:40:58.057374+0000","last_clean":"2024-03-24T11:40:58.057374+0000","last_became_active":"2024-03-24T11:39:28.577340+0000","last_became_peered":"2024-03-24T11:39:28.577340+0000","last_unstale":"2024-03-24T11:40:58.057374+0000","last_undegraded":"2024-03-24T11:40:58.057374+0000","last_fullsized":"2024-03-24T11:40:58.057374+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:12.210906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:12.210906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:12.399391+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":861,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696289+0000","last_change":"2024-03-24T11:40:20.264676+0000","last_active":"2024-03-24T11:40:59.696289+0000","last_peered":"2024-03-24T11:40:59.696289+0000","last_clean":"2024-03-24T11:40:59.696289+0000","last_became_active":"2024-03-24T11:39:28.581534+0000","last_became_peered":"2024-03-24T11:39:28.581534+0000","last_unstale":"2024-03-24T11:40:59.696289+0000","last_undegraded":"2024-03-24T11:40:59.696289+0000","last_fullsized":"2024-03-24T11:40:59.696289+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:20.263202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:40:20.263202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.422784+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":910,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.616504+0000","last_change":"2024-03-24T11:40:55.322395+0000","last_active":"2024-03-24T11:40:57.616504+0000","last_peered":"2024-03-24T11:40:57.616504+0000","last_clean":"2024-03-24T11:40:57.616504+0000","last_became_active":"2024-03-24T11:39:28.570115+0000","last_became_peered":"2024-03-24T11:39:28.570115+0000","last_unstale":"2024-03-24T11:40:57.616504+0000","last_undegraded":"2024-03-24T11:40:57.616504+0000","last_fullsized":"2024-03-24T11:40:57.616504+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:55.322293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:55.322293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:24.907856+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":948,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793814+0000","last_change":"2024-03-24T11:39:28.551439+0000","last_active":"2024-03-24T11:40:57.793814+0000","last_peered":"2024-03-24T11:40:57.793814+0000","last_clean":"2024-03-24T11:40:57.793814+0000","last_became_active":"2024-03-24T11:39:28.544005+0000","last_became_peered":"2024-03-24T11:39:28.544005+0000","last_unstale":"2024-03-24T11:40:57.793814+0000","last_undegraded":"2024-03-24T11:40:57.793814+0000","last_fullsized":"2024-03-24T11:40:57.793814+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:11.883396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.883396+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":892,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057808+0000","last_change":"2024-03-24T11:39:28.575492+0000","last_active":"2024-03-24T11:40:58.057808+0000","last_peered":"2024-03-24T11:40:58.057808+0000","last_clean":"2024-03-24T11:40:58.057808+0000","last_became_active":"2024-03-24T11:39:28.575151+0000","last_became_peered":"2024-03-24T11:39:28.575151+0000","last_unstale":"2024-03-24T11:40:58.057808+0000","last_undegraded":"2024-03-24T11:40:58.057808+0000","last_fullsized":"2024-03-24T11:40:58.057808+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:07.446672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:55.793540+0000","last_clean_scrub_stamp":"2024-03-24T11:39:07.446672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"429'9","reported_seq":1139,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696145+0000","last_change":"2024-03-24T11:39:28.596125+0000","last_active":"2024-03-24T11:40:59.696145+0000","last_peered":"2024-03-24T11:40:59.696145+0000","last_clean":"2024-03-24T11:40:59.696145+0000","last_became_active":"2024-03-24T11:39:28.581602+0000","last_became_peered":"2024-03-24T11:39:28.581602+0000","last_unstale":"2024-03-24T11:40:59.696145+0000","last_undegraded":"2024-03-24T11:40:59.696145+0000","last_fullsized":"2024-03-24T11:40:59.696145+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:54.050291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:54.050291+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":179,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.616575+0000","last_change":"2024-03-24T11:38:50.540685+0000","last_active":"2024-03-24T11:40:57.616575+0000","last_peered":"2024-03-24T11:40:57.616575+0000","last_clean":"2024-03-24T11:40:57.616575+0000","last_became_active":"2024-03-24T11:36:58.580967+0000","last_became_peered":"2024-03-24T11:36:58.580967+0000","last_unstale":"2024-03-24T11:40:57.616575+0000","last_undegraded":"2024-03-24T11:40:57.616575+0000","last_fullsized":"2024-03-24T11:40:57.616575+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:50.540148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:38:50.540148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":922,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696026+0000","last_change":"2024-03-24T11:40:12.102298+0000","last_active":"2024-03-24T11:40:59.696026+0000","last_peered":"2024-03-24T11:40:59.696026+0000","last_clean":"2024-03-24T11:40:59.696026+0000","last_became_active":"2024-03-24T11:39:28.577653+0000","last_became_peered":"2024-03-24T11:39:28.577653+0000","last_unstale":"2024-03-24T11:40:59.696026+0000","last_undegraded":"2024-03-24T11:40:59.696026+0000","last_fullsized":"2024-03-24T11:40:59.696026+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:40:12.102156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:12.102156+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:26.823746+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":179,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.615721+0000","last_change":"2024-03-24T11:39:28.567090+0000","last_active":"2024-03-24T11:40:57.615721+0000","last_peered":"2024-03-24T11:40:57.615721+0000","last_clean":"2024-03-24T11:40:57.615721+0000","last_became_active":"2024-03-24T11:39:28.566772+0000","last_became_peered":"2024-03-24T11:39:28.566772+0000","last_unstale":"2024-03-24T11:40:57.615721+0000","last_undegraded":"2024-03-24T11:40:57.615721+0000","last_fullsized":"2024-03-24T11:40:57.615721+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.220270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.220270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":1151,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793610+0000","last_change":"2024-03-24T11:38:08.961040+0000","last_active":"2024-03-24T11:40:57.793610+0000","last_peered":"2024-03-24T11:40:57.793610+0000","last_clean":"2024-03-24T11:40:57.793610+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:40:57.793610+0000","last_undegraded":"2024-03-24T11:40:57.793610+0000","last_fullsized":"2024-03-24T11:40:57.793610+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:38:08.960909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.960909+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":872,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058172+0000","last_change":"2024-03-24T11:39:28.561627+0000","last_active":"2024-03-24T11:40:58.058172+0000","last_peered":"2024-03-24T11:40:58.058172+0000","last_clean":"2024-03-24T11:40:58.058172+0000","last_became_active":"2024-03-24T11:39:28.561292+0000","last_became_peered":"2024-03-24T11:39:28.561292+0000","last_unstale":"2024-03-24T11:40:58.058172+0000","last_undegraded":"2024-03-24T11:40:58.058172+0000","last_fullsized":"2024-03-24T11:40:58.058172+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:09.604834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:09.604834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'32","reported_seq":582,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058080+0000","last_change":"2024-03-24T11:39:13.314319+0000","last_active":"2024-03-24T11:40:58.058080+0000","last_peered":"2024-03-24T11:40:58.058080+0000","last_clean":"2024-03-24T11:40:58.058080+0000","last_became_active":"2024-03-24T11:36:58.586291+0000","last_became_peered":"2024-03-24T11:36:58.586291+0000","last_unstale":"2024-03-24T11:40:58.058080+0000","last_undegraded":"2024-03-24T11:40:58.058080+0000","last_fullsized":"2024-03-24T11:40:58.058080+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:39:13.314202+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:39:13.314202+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":177,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057992+0000","last_change":"2024-03-24T11:39:28.564092+0000","last_active":"2024-03-24T11:40:58.057992+0000","last_peered":"2024-03-24T11:40:58.057992+0000","last_clean":"2024-03-24T11:40:58.057992+0000","last_became_active":"2024-03-24T11:39:28.563739+0000","last_became_peered":"2024-03-24T11:39:28.563739+0000","last_unstale":"2024-03-24T11:40:58.057992+0000","last_undegraded":"2024-03-24T11:40:58.057992+0000","last_fullsized":"2024-03-24T11:40:58.057992+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.070123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.070123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":894,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790737+0000","last_change":"2024-03-24T11:40:56.755467+0000","last_active":"2024-03-24T11:40:57.790737+0000","last_peered":"2024-03-24T11:40:57.790737+0000","last_clean":"2024-03-24T11:40:57.790737+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:40:57.790737+0000","last_undegraded":"2024-03-24T11:40:57.790737+0000","last_fullsized":"2024-03-24T11:40:57.790737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:56.755386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:40:56.755386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:22.939995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":851,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695262+0000","last_change":"2024-03-24T11:37:40.087780+0000","last_active":"2024-03-24T11:40:59.695262+0000","last_peered":"2024-03-24T11:40:59.695262+0000","last_clean":"2024-03-24T11:40:59.695262+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:40:59.695262+0000","last_undegraded":"2024-03-24T11:40:59.695262+0000","last_fullsized":"2024-03-24T11:40:59.695262+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.086223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.086223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":916,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790855+0000","last_change":"2024-03-24T11:37:55.694778+0000","last_active":"2024-03-24T11:40:57.790855+0000","last_peered":"2024-03-24T11:40:57.790855+0000","last_clean":"2024-03-24T11:40:57.790855+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:40:57.790855+0000","last_undegraded":"2024-03-24T11:40:57.790855+0000","last_fullsized":"2024-03-24T11:40:57.790855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:37:55.694628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.694628+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":495,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695737+0000","last_change":"2024-03-24T11:39:28.579890+0000","last_active":"2024-03-24T11:40:59.695737+0000","last_peered":"2024-03-24T11:40:59.695737+0000","last_clean":"2024-03-24T11:40:59.695737+0000","last_became_active":"2024-03-24T11:39:28.575621+0000","last_became_peered":"2024-03-24T11:39:28.575621+0000","last_unstale":"2024-03-24T11:40:59.695737+0000","last_undegraded":"2024-03-24T11:40:59.695737+0000","last_fullsized":"2024-03-24T11:40:59.695737+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.172813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.172813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":855,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695781+0000","last_change":"2024-03-24T11:39:15.574465+0000","last_active":"2024-03-24T11:40:59.695781+0000","last_peered":"2024-03-24T11:40:59.695781+0000","last_clean":"2024-03-24T11:40:59.695781+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:40:59.695781+0000","last_undegraded":"2024-03-24T11:40:59.695781+0000","last_fullsized":"2024-03-24T11:40:59.695781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:15.574327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:15.574327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":874,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057343+0000","last_change":"2024-03-24T11:40:09.537029+0000","last_active":"2024-03-24T11:40:58.057343+0000","last_peered":"2024-03-24T11:40:58.057343+0000","last_clean":"2024-03-24T11:40:58.057343+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:40:58.057343+0000","last_undegraded":"2024-03-24T11:40:58.057343+0000","last_fullsized":"2024-03-24T11:40:58.057343+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:09.536910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:09.536910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.356810+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":920,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695460+0000","last_change":"2024-03-24T11:40:24.161914+0000","last_active":"2024-03-24T11:40:59.695460+0000","last_peered":"2024-03-24T11:40:59.695460+0000","last_clean":"2024-03-24T11:40:59.695460+0000","last_became_active":"2024-03-24T11:39:28.575371+0000","last_became_peered":"2024-03-24T11:39:28.575371+0000","last_unstale":"2024-03-24T11:40:59.695460+0000","last_undegraded":"2024-03-24T11:40:59.695460+0000","last_fullsized":"2024-03-24T11:40:59.695460+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:40:24.161610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:24.161610+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:48.775035+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":842,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791397+0000","last_change":"2024-03-24T11:40:35.292266+0000","last_active":"2024-03-24T11:40:57.791397+0000","last_peered":"2024-03-24T11:40:57.791397+0000","last_clean":"2024-03-24T11:40:57.791397+0000","last_became_active":"2024-03-24T11:40:35.291627+0000","last_became_peered":"2024-03-24T11:40:35.291627+0000","last_unstale":"2024-03-24T11:40:57.791397+0000","last_undegraded":"2024-03-24T11:40:57.791397+0000","last_fullsized":"2024-03-24T11:40:57.791397+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.913497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.913497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":187,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057253+0000","last_change":"2024-03-24T11:39:28.552174+0000","last_active":"2024-03-24T11:40:58.057253+0000","last_peered":"2024-03-24T11:40:58.057253+0000","last_clean":"2024-03-24T11:40:58.057253+0000","last_became_active":"2024-03-24T11:39:28.551550+0000","last_became_peered":"2024-03-24T11:39:28.551550+0000","last_unstale":"2024-03-24T11:40:58.057253+0000","last_undegraded":"2024-03-24T11:40:58.057253+0000","last_fullsized":"2024-03-24T11:40:58.057253+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_clean_scrub_stamp":"2024-03-24T11:39:06.116941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":851,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696008+0000","last_change":"2024-03-24T11:36:48.160280+0000","last_active":"2024-03-24T11:40:59.696008+0000","last_peered":"2024-03-24T11:40:59.696008+0000","last_clean":"2024-03-24T11:40:59.696008+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:40:59.696008+0000","last_undegraded":"2024-03-24T11:40:59.696008+0000","last_fullsized":"2024-03-24T11:40:59.696008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:48.160164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:48.160164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"421'14","reported_seq":879,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695254+0000","last_change":"2024-03-24T11:38:54.224427+0000","last_active":"2024-03-24T11:40:59.695254+0000","last_peered":"2024-03-24T11:40:59.695254+0000","last_clean":"2024-03-24T11:40:59.695254+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:40:59.695254+0000","last_undegraded":"2024-03-24T11:40:59.695254+0000","last_fullsized":"2024-03-24T11:40:59.695254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:38:54.222100+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:38:54.222100+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":886,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058332+0000","last_change":"2024-03-24T11:38:53.264039+0000","last_active":"2024-03-24T11:40:58.058332+0000","last_peered":"2024-03-24T11:40:58.058332+0000","last_clean":"2024-03-24T11:40:58.058332+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:40:58.058332+0000","last_undegraded":"2024-03-24T11:40:58.058332+0000","last_fullsized":"2024-03-24T11:40:58.058332+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.263761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.263761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":906,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057602+0000","last_change":"2024-03-24T11:39:28.572502+0000","last_active":"2024-03-24T11:40:58.057602+0000","last_peered":"2024-03-24T11:40:58.057602+0000","last_clean":"2024-03-24T11:40:58.057602+0000","last_became_active":"2024-03-24T11:39:28.572151+0000","last_became_peered":"2024-03-24T11:39:28.572151+0000","last_unstale":"2024-03-24T11:40:58.057602+0000","last_undegraded":"2024-03-24T11:40:58.057602+0000","last_fullsized":"2024-03-24T11:40:58.057602+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.997698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.997698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":178,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695222+0000","last_change":"2024-03-24T11:39:28.588074+0000","last_active":"2024-03-24T11:40:59.695222+0000","last_peered":"2024-03-24T11:40:59.695222+0000","last_clean":"2024-03-24T11:40:59.695222+0000","last_became_active":"2024-03-24T11:39:28.585224+0000","last_became_peered":"2024-03-24T11:39:28.585224+0000","last_unstale":"2024-03-24T11:40:59.695222+0000","last_undegraded":"2024-03-24T11:40:59.695222+0000","last_fullsized":"2024-03-24T11:40:59.695222+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:06.147813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:06.147813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"421'32","reported_seq":955,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058538+0000","last_change":"2024-03-24T11:40:56.233621+0000","last_active":"2024-03-24T11:40:58.058538+0000","last_peered":"2024-03-24T11:40:58.058538+0000","last_clean":"2024-03-24T11:40:58.058538+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:40:58.058538+0000","last_undegraded":"2024-03-24T11:40:58.058538+0000","last_fullsized":"2024-03-24T11:40:58.058538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:40:56.233511+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:40:56.233511+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:22.918300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":863,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695191+0000","last_change":"2024-03-24T11:38:01.211155+0000","last_active":"2024-03-24T11:40:59.695191+0000","last_peered":"2024-03-24T11:40:59.695191+0000","last_clean":"2024-03-24T11:40:59.695191+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:40:59.695191+0000","last_undegraded":"2024-03-24T11:40:59.695191+0000","last_fullsized":"2024-03-24T11:40:59.695191+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.211052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.211052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":885,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058504+0000","last_change":"2024-03-24T11:40:23.307812+0000","last_active":"2024-03-24T11:40:58.058504+0000","last_peered":"2024-03-24T11:40:58.058504+0000","last_clean":"2024-03-24T11:40:58.058504+0000","last_became_active":"2024-03-24T11:39:28.576207+0000","last_became_peered":"2024-03-24T11:39:28.576207+0000","last_unstale":"2024-03-24T11:40:58.058504+0000","last_undegraded":"2024-03-24T11:40:58.058504+0000","last_fullsized":"2024-03-24T11:40:58.058504+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:23.307700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:40:23.307700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.835484+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":887,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791261+0000","last_change":"2024-03-24T11:38:12.027553+0000","last_active":"2024-03-24T11:40:57.791261+0000","last_peered":"2024-03-24T11:40:57.791261+0000","last_clean":"2024-03-24T11:40:57.791261+0000","last_became_active":"2024-03-24T11:36:58.592355+0000","last_became_peered":"2024-03-24T11:36:58.592355+0000","last_unstale":"2024-03-24T11:40:57.791261+0000","last_undegraded":"2024-03-24T11:40:57.791261+0000","last_fullsized":"2024-03-24T11:40:57.791261+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.027430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'30","reported_seq":972,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058386+0000","last_change":"2024-03-24T11:40:15.366849+0000","last_active":"2024-03-24T11:40:58.058386+0000","last_peered":"2024-03-24T11:40:58.058386+0000","last_clean":"2024-03-24T11:40:58.058386+0000","last_became_active":"2024-03-24T11:39:28.576661+0000","last_became_peered":"2024-03-24T11:39:28.576661+0000","last_unstale":"2024-03-24T11:40:58.058386+0000","last_undegraded":"2024-03-24T11:40:58.058386+0000","last_fullsized":"2024-03-24T11:40:58.058386+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:40:15.366706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.366706+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:26.739599+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":905,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790549+0000","last_change":"2024-03-24T11:37:52.677447+0000","last_active":"2024-03-24T11:40:57.790549+0000","last_peered":"2024-03-24T11:40:57.790549+0000","last_clean":"2024-03-24T11:40:57.790549+0000","last_became_active":"2024-03-24T11:36:58.593112+0000","last_became_peered":"2024-03-24T11:36:58.593112+0000","last_unstale":"2024-03-24T11:40:57.790549+0000","last_undegraded":"2024-03-24T11:40:57.790549+0000","last_fullsized":"2024-03-24T11:40:57.790549+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:37:52.677294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.677294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"429'11","reported_seq":1203,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058441+0000","last_change":"2024-03-24T11:40:52.183243+0000","last_active":"2024-03-24T11:40:58.058441+0000","last_peered":"2024-03-24T11:40:58.058441+0000","last_clean":"2024-03-24T11:40:58.058441+0000","last_became_active":"2024-03-24T11:39:28.579771+0000","last_became_peered":"2024-03-24T11:39:28.579771+0000","last_unstale":"2024-03-24T11:40:58.058441+0000","last_undegraded":"2024-03-24T11:40:58.058441+0000","last_fullsized":"2024-03-24T11:40:58.058441+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'11","last_scrub_stamp":"2024-03-24T11:40:52.183120+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:40:52.183120+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:11.717493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":547,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793153+0000","last_change":"2024-03-24T11:39:28.582106+0000","last_active":"2024-03-24T11:40:57.793153+0000","last_peered":"2024-03-24T11:40:57.793153+0000","last_clean":"2024-03-24T11:40:57.793153+0000","last_became_active":"2024-03-24T11:39:28.562081+0000","last_became_peered":"2024-03-24T11:39:28.562081+0000","last_unstale":"2024-03-24T11:40:57.793153+0000","last_undegraded":"2024-03-24T11:40:57.793153+0000","last_fullsized":"2024-03-24T11:40:57.793153+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:39:19.096734+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:39:19.096734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"421'24","reported_seq":909,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695947+0000","last_change":"2024-03-24T11:36:59.961408+0000","last_active":"2024-03-24T11:40:59.695947+0000","last_peered":"2024-03-24T11:40:59.695947+0000","last_clean":"2024-03-24T11:40:59.695947+0000","last_became_active":"2024-03-24T11:36:58.591714+0000","last_became_peered":"2024-03-24T11:36:58.591714+0000","last_unstale":"2024-03-24T11:40:59.695947+0000","last_undegraded":"2024-03-24T11:40:59.695947+0000","last_fullsized":"2024-03-24T11:40:59.695947+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:36:59.961306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:59.961306+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":171,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790660+0000","last_change":"2024-03-24T11:37:50.620288+0000","last_active":"2024-03-24T11:40:57.790660+0000","last_peered":"2024-03-24T11:40:57.790660+0000","last_clean":"2024-03-24T11:40:57.790660+0000","last_became_active":"2024-03-24T11:36:58.597448+0000","last_became_peered":"2024-03-24T11:36:58.597448+0000","last_unstale":"2024-03-24T11:40:57.790660+0000","last_undegraded":"2024-03-24T11:40:57.790660+0000","last_fullsized":"2024-03-24T11:40:57.790660+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:50.620171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:50.620171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":180,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696056+0000","last_change":"2024-03-24T11:39:28.581090+0000","last_active":"2024-03-24T11:40:59.696056+0000","last_peered":"2024-03-24T11:40:59.696056+0000","last_clean":"2024-03-24T11:40:59.696056+0000","last_became_active":"2024-03-24T11:39:28.577496+0000","last_became_peered":"2024-03-24T11:39:28.577496+0000","last_unstale":"2024-03-24T11:40:59.696056+0000","last_undegraded":"2024-03-24T11:40:59.696056+0000","last_fullsized":"2024-03-24T11:40:59.696056+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.066063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.066063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":855,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696063+0000","last_change":"2024-03-24T11:40:54.412222+0000","last_active":"2024-03-24T11:40:59.696063+0000","last_peered":"2024-03-24T11:40:59.696063+0000","last_clean":"2024-03-24T11:40:59.696063+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:40:59.696063+0000","last_undegraded":"2024-03-24T11:40:59.696063+0000","last_fullsized":"2024-03-24T11:40:59.696063+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:54.412091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:54.412091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:11.778682+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"421'39","reported_seq":966,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.616208+0000","last_change":"2024-03-24T11:39:15.286788+0000","last_active":"2024-03-24T11:40:57.616208+0000","last_peered":"2024-03-24T11:40:57.616208+0000","last_clean":"2024-03-24T11:40:57.616208+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:40:57.616208+0000","last_undegraded":"2024-03-24T11:40:57.616208+0000","last_fullsized":"2024-03-24T11:40:57.616208+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:39:15.286646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:15.286646+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":897,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057944+0000","last_change":"2024-03-24T11:39:28.573154+0000","last_active":"2024-03-24T11:40:58.057944+0000","last_peered":"2024-03-24T11:40:58.057944+0000","last_clean":"2024-03-24T11:40:58.057944+0000","last_became_active":"2024-03-24T11:39:28.572672+0000","last_became_peered":"2024-03-24T11:39:28.572672+0000","last_unstale":"2024-03-24T11:40:58.057944+0000","last_undegraded":"2024-03-24T11:40:58.057944+0000","last_fullsized":"2024-03-24T11:40:58.057944+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:05.380874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:39:05.380874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":883,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.791158+0000","last_change":"2024-03-24T11:37:43.684412+0000","last_active":"2024-03-24T11:40:57.791158+0000","last_peered":"2024-03-24T11:40:57.791158+0000","last_clean":"2024-03-24T11:40:57.791158+0000","last_became_active":"2024-03-24T11:36:58.596592+0000","last_became_peered":"2024-03-24T11:36:58.596592+0000","last_unstale":"2024-03-24T11:40:57.791158+0000","last_undegraded":"2024-03-24T11:40:57.791158+0000","last_fullsized":"2024-03-24T11:40:57.791158+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.684286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.684286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":180,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058412+0000","last_change":"2024-03-24T11:40:16.249656+0000","last_active":"2024-03-24T11:40:58.058412+0000","last_peered":"2024-03-24T11:40:58.058412+0000","last_clean":"2024-03-24T11:40:58.058412+0000","last_became_active":"2024-03-24T11:36:58.575356+0000","last_became_peered":"2024-03-24T11:36:58.575356+0000","last_unstale":"2024-03-24T11:40:58.058412+0000","last_undegraded":"2024-03-24T11:40:58.058412+0000","last_fullsized":"2024-03-24T11:40:58.058412+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:16.249562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:40:16.249562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:34.813107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"421'9","reported_seq":530,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695604+0000","last_change":"2024-03-24T11:36:58.602264+0000","last_active":"2024-03-24T11:40:59.695604+0000","last_peered":"2024-03-24T11:40:59.695604+0000","last_clean":"2024-03-24T11:40:59.695604+0000","last_became_active":"2024-03-24T11:36:58.586221+0000","last_became_peered":"2024-03-24T11:36:58.586221+0000","last_unstale":"2024-03-24T11:40:59.695604+0000","last_undegraded":"2024-03-24T11:40:59.695604+0000","last_fullsized":"2024-03-24T11:40:59.695604+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'7","last_scrub_stamp":"2024-03-24T11:36:04.265719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:04.265719+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":891,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793409+0000","last_change":"2024-03-24T11:40:24.196213+0000","last_active":"2024-03-24T11:40:57.793409+0000","last_peered":"2024-03-24T11:40:57.793409+0000","last_clean":"2024-03-24T11:40:57.793409+0000","last_became_active":"2024-03-24T11:36:58.585301+0000","last_became_peered":"2024-03-24T11:36:58.585301+0000","last_unstale":"2024-03-24T11:40:57.793409+0000","last_undegraded":"2024-03-24T11:40:57.793409+0000","last_fullsized":"2024-03-24T11:40:57.793409+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:24.196131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:40:24.196131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.249324+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":187,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.790587+0000","last_change":"2024-03-24T11:40:11.727255+0000","last_active":"2024-03-24T11:40:57.790587+0000","last_peered":"2024-03-24T11:40:57.790587+0000","last_clean":"2024-03-24T11:40:57.790587+0000","last_became_active":"2024-03-24T11:39:28.571768+0000","last_became_peered":"2024-03-24T11:39:28.571768+0000","last_unstale":"2024-03-24T11:40:57.790587+0000","last_undegraded":"2024-03-24T11:40:57.790587+0000","last_fullsized":"2024-03-24T11:40:57.790587+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:11.727141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.727141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:29.376135+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":874,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792989+0000","last_change":"2024-03-24T11:40:54.116829+0000","last_active":"2024-03-24T11:40:57.792989+0000","last_peered":"2024-03-24T11:40:57.792989+0000","last_clean":"2024-03-24T11:40:57.792989+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:40:57.792989+0000","last_undegraded":"2024-03-24T11:40:57.792989+0000","last_fullsized":"2024-03-24T11:40:57.792989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:54.116658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:40:54.116658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:15.911672+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"421'21","reported_seq":552,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695495+0000","last_change":"2024-03-24T11:38:53.240408+0000","last_active":"2024-03-24T11:40:59.695495+0000","last_peered":"2024-03-24T11:40:59.695495+0000","last_clean":"2024-03-24T11:40:59.695495+0000","last_became_active":"2024-03-24T11:36:58.580893+0000","last_became_peered":"2024-03-24T11:36:58.580893+0000","last_unstale":"2024-03-24T11:40:59.695495+0000","last_undegraded":"2024-03-24T11:40:59.695495+0000","last_fullsized":"2024-03-24T11:40:59.695495+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:53.238808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.238808+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":891,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.793594+0000","last_change":"2024-03-24T11:39:07.004625+0000","last_active":"2024-03-24T11:40:57.793594+0000","last_peered":"2024-03-24T11:40:57.793594+0000","last_clean":"2024-03-24T11:40:57.793594+0000","last_became_active":"2024-03-24T11:36:58.590154+0000","last_became_peered":"2024-03-24T11:36:58.590154+0000","last_unstale":"2024-03-24T11:40:57.793594+0000","last_undegraded":"2024-03-24T11:40:57.793594+0000","last_fullsized":"2024-03-24T11:40:57.793594+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:07.004520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:39:07.004520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":897,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792900+0000","last_change":"2024-03-24T11:40:55.105221+0000","last_active":"2024-03-24T11:40:57.792900+0000","last_peered":"2024-03-24T11:40:57.792900+0000","last_clean":"2024-03-24T11:40:57.792900+0000","last_became_active":"2024-03-24T11:39:28.562146+0000","last_became_peered":"2024-03-24T11:39:28.562146+0000","last_unstale":"2024-03-24T11:40:57.792900+0000","last_undegraded":"2024-03-24T11:40:57.792900+0000","last_fullsized":"2024-03-24T11:40:57.792900+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:55.105042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:40:55.105042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:59.762454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":880,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057934+0000","last_change":"2024-03-24T11:40:17.412824+0000","last_active":"2024-03-24T11:40:58.057934+0000","last_peered":"2024-03-24T11:40:58.057934+0000","last_clean":"2024-03-24T11:40:58.057934+0000","last_became_active":"2024-03-24T11:39:28.548958+0000","last_became_peered":"2024-03-24T11:39:28.548958+0000","last_unstale":"2024-03-24T11:40:58.057934+0000","last_undegraded":"2024-03-24T11:40:58.057934+0000","last_fullsized":"2024-03-24T11:40:58.057934+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:17.412643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.412643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:31.388086+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'39","reported_seq":912,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.696233+0000","last_change":"2024-03-24T11:37:02.002116+0000","last_active":"2024-03-24T11:40:59.696233+0000","last_peered":"2024-03-24T11:40:59.696233+0000","last_clean":"2024-03-24T11:40:59.696233+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:40:59.696233+0000","last_undegraded":"2024-03-24T11:40:59.696233+0000","last_fullsized":"2024-03-24T11:40:59.696233+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:37:02.002019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:02.002019+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":857,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695991+0000","last_change":"2024-03-24T11:35:51.761976+0000","last_active":"2024-03-24T11:40:59.695991+0000","last_peered":"2024-03-24T11:40:59.695991+0000","last_clean":"2024-03-24T11:40:59.695991+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:40:59.695991+0000","last_undegraded":"2024-03-24T11:40:59.695991+0000","last_fullsized":"2024-03-24T11:40:59.695991+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:35:51.761850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:35:51.761850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":180,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057397+0000","last_change":"2024-03-24T11:39:11.310606+0000","last_active":"2024-03-24T11:40:58.057397+0000","last_peered":"2024-03-24T11:40:58.057397+0000","last_clean":"2024-03-24T11:40:58.057397+0000","last_became_active":"2024-03-24T11:36:58.584540+0000","last_became_peered":"2024-03-24T11:36:58.584540+0000","last_unstale":"2024-03-24T11:40:58.057397+0000","last_undegraded":"2024-03-24T11:40:58.057397+0000","last_fullsized":"2024-03-24T11:40:58.057397+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:11.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:11.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":848,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-24T11:40:59.695856+0000","last_change":"2024-03-24T11:39:28.584057+0000","last_active":"2024-03-24T11:40:59.695856+0000","last_peered":"2024-03-24T11:40:59.695856+0000","last_clean":"2024-03-24T11:40:59.695856+0000","last_became_active":"2024-03-24T11:39:28.580757+0000","last_became_peered":"2024-03-24T11:39:28.580757+0000","last_unstale":"2024-03-24T11:40:59.695856+0000","last_undegraded":"2024-03-24T11:40:59.695856+0000","last_fullsized":"2024-03-24T11:40:59.695856+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:53.164514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:53.164514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":582,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.057658+0000","last_change":"2024-03-24T11:39:17.302385+0000","last_active":"2024-03-24T11:40:58.057658+0000","last_peered":"2024-03-24T11:40:58.057658+0000","last_clean":"2024-03-24T11:40:58.057658+0000","last_became_active":"2024-03-24T11:36:58.580796+0000","last_became_peered":"2024-03-24T11:36:58.580796+0000","last_unstale":"2024-03-24T11:40:58.057658+0000","last_undegraded":"2024-03-24T11:40:58.057658+0000","last_fullsized":"2024-03-24T11:40:58.057658+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:39:17.302257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:17.302257+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":887,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:58.058224+0000","last_change":"2024-03-24T11:40:25.277893+0000","last_active":"2024-03-24T11:40:58.058224+0000","last_peered":"2024-03-24T11:40:58.058224+0000","last_clean":"2024-03-24T11:40:58.058224+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:40:58.058224+0000","last_undegraded":"2024-03-24T11:40:58.058224+0000","last_fullsized":"2024-03-24T11:40:58.058224+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:25.277603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:25.277603+0000","last_clean_scrub_stamp":"2024-03-24T11:40:25.277603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:50.379971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":892,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792120+0000","last_change":"2024-03-24T11:40:57.792140+0000","last_active":"2024-03-24T11:40:57.792120+0000","last_peered":"2024-03-24T11:40:57.792120+0000","last_clean":"2024-03-24T11:40:57.792120+0000","last_became_active":"2024-03-24T11:39:28.557876+0000","last_became_peered":"2024-03-24T11:39:28.557876+0000","last_unstale":"2024-03-24T11:40:57.792120+0000","last_undegraded":"2024-03-24T11:40:57.792120+0000","last_fullsized":"2024-03-24T11:40:57.792120+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:57.792057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:57.792057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:27.070962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":185,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-24T11:40:57.792878+0000","last_change":"2024-03-24T11:40:11.312058+0000","last_active":"2024-03-24T11:40:57.792878+0000","last_peered":"2024-03-24T11:40:57.792878+0000","last_clean":"2024-03-24T11:40:57.792878+0000","last_became_active":"2024-03-24T11:39:28.546592+0000","last_became_peered":"2024-03-24T11:39:28.546592+0000","last_unstale":"2024-03-24T11:40:57.792878+0000","last_undegraded":"2024-03-24T11:40:57.792878+0000","last_fullsized":"2024-03-24T11:40:57.792878+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:11.311949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.311949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:20.458238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":623,"num_read_kb":618,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":5568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":3856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":13944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1031,"ondisk_log_size":1031,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070733,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365074,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":730672,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":93014480,"statfs":{"total":95995035648,"available":95246827520,"internally_reserved":0,"allocated":1216512,"data_stored":836378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":746977734},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934472,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92810680,"statfs":{"total":95995035648,"available":95038136320,"internally_reserved":0,"allocated":917504,"data_stored":501495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738673,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840048,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92905104,"statfs":{"total":95995035648,"available":95134826496,"internally_reserved":0,"allocated":888832,"data_stored":507790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411623,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722284,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93022868,"statfs":{"total":95995035648,"available":95255416832,"internally_reserved":0,"allocated":1146880,"data_stored":770305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149984,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880916,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864236,"statfs":{"total":95995035648,"available":95092977664,"internally_reserved":0,"allocated":1515520,"data_stored":1108133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":900528583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659415,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848968,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896184,"statfs":{"total":95995035648,"available":95125692416,"internally_reserved":0,"allocated":1503232,"data_stored":1098857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:41:05.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:41:05.350 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: from='client.16059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:41:05.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:05.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:05 smithi181 ceph-mon[33998]: pgmap v1031: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:41:05.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:41:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: from='client.16059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[37907]: pgmap v1031: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:41:05.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:05.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:05.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: from='client.16059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:41:05.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:41:05.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:41:05.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:05.461 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:05 smithi179 ceph-mon[29608]: pgmap v1031: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T11:41:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:06 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:06 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:06 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:41:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:06 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:41:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:06 smithi181 ceph-mon[33998]: 3.1e scrub starts 2024-03-24T11:41:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:06 smithi181 ceph-mon[33998]: 3.1e scrub ok 2024-03-24T11:41:06.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:06 smithi181 ceph-mon[33998]: osdmap e466: 8 total, 8 up, 6 in 2024-03-24T11:41:06.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:06.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:06.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[29608]: 3.1e scrub starts 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[29608]: 3.1e scrub ok 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[29608]: osdmap e466: 8 total, 8 up, 6 in 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[37907]: 3.1e scrub starts 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[37907]: 3.1e scrub ok 2024-03-24T11:41:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:06 smithi179 ceph-mon[37907]: osdmap e466: 8 total, 8 up, 6 in 2024-03-24T11:41:07.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:07 smithi179 ceph-mon[29608]: 4.16 deep-scrub starts 2024-03-24T11:41:07.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:07 smithi179 ceph-mon[29608]: 4.16 deep-scrub ok 2024-03-24T11:41:07.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:07 smithi179 ceph-mon[29608]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 744 B/s rd, 0 op/s 2024-03-24T11:41:07.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:07 smithi179 ceph-mon[37907]: 4.16 deep-scrub starts 2024-03-24T11:41:07.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:07 smithi179 ceph-mon[37907]: 4.16 deep-scrub ok 2024-03-24T11:41:07.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:07 smithi179 ceph-mon[37907]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 744 B/s rd, 0 op/s 2024-03-24T11:41:07.331 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:41:07.335 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:41:07.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:07 smithi181 ceph-mon[33998]: 4.16 deep-scrub starts 2024-03-24T11:41:07.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:07 smithi181 ceph-mon[33998]: 4.16 deep-scrub ok 2024-03-24T11:41:07.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:07 smithi181 ceph-mon[33998]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 744 B/s rd, 0 op/s 2024-03-24T11:41:08.119 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":1033,"stamp":"2024-03-24T11:41:06.504329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2198,"num_read_kb":1950,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":661172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1512,"ondisk_log_size":1512,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4957488,"kb_used_data":7152,"kb_used_omap":9,"kb_used_meta":4950262,"kb_avail":557513424,"statfs":{"total":575970213888,"available":570893746176,"internally_reserved":0,"allocated":7323648,"data_stored":4910734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":5069068961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.877124"},"pg_stats":[{"pgid":"4.c","version":"421'30","reported_seq":569,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386398+0000","last_change":"2024-03-24T11:38:10.116849+0000","last_active":"2024-03-24T11:41:05.386398+0000","last_peered":"2024-03-24T11:41:05.386398+0000","last_clean":"2024-03-24T11:41:05.386398+0000","last_became_active":"2024-03-24T11:36:58.591623+0000","last_became_peered":"2024-03-24T11:36:58.591623+0000","last_unstale":"2024-03-24T11:41:05.386398+0000","last_undegraded":"2024-03-24T11:41:05.386398+0000","last_fullsized":"2024-03-24T11:41:05.386398+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:10.116699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:10.116699+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":873,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.385925+0000","last_change":"2024-03-24T11:40:15.063982+0000","last_active":"2024-03-24T11:41:05.385925+0000","last_peered":"2024-03-24T11:41:05.385925+0000","last_clean":"2024-03-24T11:41:05.385925+0000","last_became_active":"2024-03-24T11:36:58.586602+0000","last_became_peered":"2024-03-24T11:36:58.586602+0000","last_unstale":"2024-03-24T11:41:05.385925+0000","last_undegraded":"2024-03-24T11:41:05.385925+0000","last_fullsized":"2024-03-24T11:41:05.385925+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:15.063776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.063776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:15.414894+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"429'10","reported_seq":1199,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.306300+0000","last_change":"2024-03-24T11:41:00.234454+0000","last_active":"2024-03-24T11:41:03.306300+0000","last_peered":"2024-03-24T11:41:03.306300+0000","last_clean":"2024-03-24T11:41:03.306300+0000","last_became_active":"2024-03-24T11:39:28.568543+0000","last_became_peered":"2024-03-24T11:39:28.568543+0000","last_unstale":"2024-03-24T11:41:03.306300+0000","last_undegraded":"2024-03-24T11:41:03.306300+0000","last_fullsized":"2024-03-24T11:41:03.306300+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'10","last_scrub_stamp":"2024-03-24T11:41:00.234369+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:41:00.234369+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:24.391780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":175,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327437+0000","last_change":"2024-03-24T11:38:01.792439+0000","last_active":"2024-03-24T11:41:03.327437+0000","last_peered":"2024-03-24T11:41:03.327437+0000","last_clean":"2024-03-24T11:41:03.327437+0000","last_became_active":"2024-03-24T11:36:58.590765+0000","last_became_peered":"2024-03-24T11:36:58.590765+0000","last_unstale":"2024-03-24T11:41:03.327437+0000","last_undegraded":"2024-03-24T11:41:03.327437+0000","last_fullsized":"2024-03-24T11:41:03.327437+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.792132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:39.980621+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.792132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":952,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.384051+0000","last_change":"2024-03-24T11:39:31.408536+0000","last_active":"2024-03-24T11:41:05.384051+0000","last_peered":"2024-03-24T11:41:05.384051+0000","last_clean":"2024-03-24T11:41:05.384051+0000","last_became_active":"2024-03-24T11:39:31.407869+0000","last_became_peered":"2024-03-24T11:39:31.407869+0000","last_unstale":"2024-03-24T11:41:05.384051+0000","last_undegraded":"2024-03-24T11:41:05.384051+0000","last_fullsized":"2024-03-24T11:41:05.384051+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:36:57.972898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:36:57.972898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"421'53","reported_seq":654,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.306183+0000","last_change":"2024-03-24T11:40:16.310832+0000","last_active":"2024-03-24T11:41:03.306183+0000","last_peered":"2024-03-24T11:41:03.306183+0000","last_clean":"2024-03-24T11:41:03.306183+0000","last_became_active":"2024-03-24T11:36:58.580651+0000","last_became_peered":"2024-03-24T11:36:58.580651+0000","last_unstale":"2024-03-24T11:41:03.306183+0000","last_undegraded":"2024-03-24T11:41:03.306183+0000","last_fullsized":"2024-03-24T11:41:03.306183+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'53","last_scrub_stamp":"2024-03-24T11:40:16.310682+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:40:16.310682+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.329404+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":924,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260520+0000","last_change":"2024-03-24T11:38:48.018756+0000","last_active":"2024-03-24T11:41:03.260520+0000","last_peered":"2024-03-24T11:41:03.260520+0000","last_clean":"2024-03-24T11:41:03.260520+0000","last_became_active":"2024-03-24T11:36:58.587097+0000","last_became_peered":"2024-03-24T11:36:58.587097+0000","last_unstale":"2024-03-24T11:41:03.260520+0000","last_undegraded":"2024-03-24T11:41:03.260520+0000","last_fullsized":"2024-03-24T11:41:03.260520+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:38:48.018617+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:38:48.018617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":896,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327437+0000","last_change":"2024-03-24T11:40:18.788766+0000","last_active":"2024-03-24T11:41:03.327437+0000","last_peered":"2024-03-24T11:41:03.327437+0000","last_clean":"2024-03-24T11:41:03.327437+0000","last_became_active":"2024-03-24T11:36:58.595572+0000","last_became_peered":"2024-03-24T11:36:58.595572+0000","last_unstale":"2024-03-24T11:41:03.327437+0000","last_undegraded":"2024-03-24T11:41:03.327437+0000","last_fullsized":"2024-03-24T11:41:03.327437+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:18.788516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:18.788516+0000","last_clean_scrub_stamp":"2024-03-24T11:40:18.788516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.145429+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":201,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386772+0000","last_change":"2024-03-24T11:38:52.569860+0000","last_active":"2024-03-24T11:41:05.386772+0000","last_peered":"2024-03-24T11:41:05.386772+0000","last_clean":"2024-03-24T11:41:05.386772+0000","last_became_active":"2024-03-24T11:36:58.582225+0000","last_became_peered":"2024-03-24T11:36:58.582225+0000","last_unstale":"2024-03-24T11:41:05.386772+0000","last_undegraded":"2024-03-24T11:41:05.386772+0000","last_fullsized":"2024-03-24T11:41:05.386772+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:52.569730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.569730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":1047,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852220+0000","last_change":"2024-03-24T11:38:03.034263+0000","last_active":"2024-03-24T11:41:02.852220+0000","last_peered":"2024-03-24T11:41:02.852220+0000","last_clean":"2024-03-24T11:41:02.852220+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:41:02.852220+0000","last_undegraded":"2024-03-24T11:41:02.852220+0000","last_fullsized":"2024-03-24T11:41:02.852220+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:38:03.034131+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:38:03.034131+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":581,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261143+0000","last_change":"2024-03-24T11:40:17.292448+0000","last_active":"2024-03-24T11:41:03.261143+0000","last_peered":"2024-03-24T11:41:03.261143+0000","last_clean":"2024-03-24T11:41:03.261143+0000","last_became_active":"2024-03-24T11:36:58.587116+0000","last_became_peered":"2024-03-24T11:36:58.587116+0000","last_unstale":"2024-03-24T11:41:03.261143+0000","last_undegraded":"2024-03-24T11:41:03.261143+0000","last_fullsized":"2024-03-24T11:41:03.261143+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:40:17.291968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.291968+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:43.826573+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":189,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.305686+0000","last_change":"2024-03-24T11:40:17.263847+0000","last_active":"2024-03-24T11:41:03.305686+0000","last_peered":"2024-03-24T11:41:03.305686+0000","last_clean":"2024-03-24T11:41:03.305686+0000","last_became_active":"2024-03-24T11:39:28.553193+0000","last_became_peered":"2024-03-24T11:39:28.553193+0000","last_unstale":"2024-03-24T11:41:03.305686+0000","last_undegraded":"2024-03-24T11:41:03.305686+0000","last_fullsized":"2024-03-24T11:41:03.305686+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:17.263730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.263730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:19.660517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":888,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853609+0000","last_change":"2024-03-24T11:40:59.104752+0000","last_active":"2024-03-24T11:41:02.853609+0000","last_peered":"2024-03-24T11:41:02.853609+0000","last_clean":"2024-03-24T11:41:02.853609+0000","last_became_active":"2024-03-24T11:36:58.575297+0000","last_became_peered":"2024-03-24T11:36:58.575297+0000","last_unstale":"2024-03-24T11:41:02.853609+0000","last_undegraded":"2024-03-24T11:41:02.853609+0000","last_fullsized":"2024-03-24T11:41:02.853609+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:59.104606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:40:59.104606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:15.675251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":855,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386272+0000","last_change":"2024-03-24T11:40:17.160060+0000","last_active":"2024-03-24T11:41:05.386272+0000","last_peered":"2024-03-24T11:41:05.386272+0000","last_clean":"2024-03-24T11:41:05.386272+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:41:05.386272+0000","last_undegraded":"2024-03-24T11:41:05.386272+0000","last_fullsized":"2024-03-24T11:41:05.386272+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:17.159943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.159943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:30.247287+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"421'46","reported_seq":1006,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327430+0000","last_change":"2024-03-24T11:40:20.669735+0000","last_active":"2024-03-24T11:41:03.327430+0000","last_peered":"2024-03-24T11:41:03.327430+0000","last_clean":"2024-03-24T11:41:03.327430+0000","last_became_active":"2024-03-24T11:39:28.585895+0000","last_became_peered":"2024-03-24T11:39:28.585895+0000","last_unstale":"2024-03-24T11:41:03.327430+0000","last_undegraded":"2024-03-24T11:41:03.327430+0000","last_fullsized":"2024-03-24T11:41:03.327430+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-24T11:40:20.669353+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:40:20.669353+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:22.302647+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":874,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.384045+0000","last_change":"2024-03-24T11:39:28.589571+0000","last_active":"2024-03-24T11:41:05.384045+0000","last_peered":"2024-03-24T11:41:05.384045+0000","last_clean":"2024-03-24T11:41:05.384045+0000","last_became_active":"2024-03-24T11:39:28.587712+0000","last_became_peered":"2024-03-24T11:39:28.587712+0000","last_unstale":"2024-03-24T11:41:05.384045+0000","last_undegraded":"2024-03-24T11:41:05.384045+0000","last_fullsized":"2024-03-24T11:41:05.384045+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:01.005586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:37:01.005586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'9","reported_seq":261,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853331+0000","last_change":"2024-03-24T11:38:51.903763+0000","last_active":"2024-03-24T11:41:02.853331+0000","last_peered":"2024-03-24T11:41:02.853331+0000","last_clean":"2024-03-24T11:41:02.853331+0000","last_became_active":"2024-03-24T11:36:58.574227+0000","last_became_peered":"2024-03-24T11:36:58.574227+0000","last_unstale":"2024-03-24T11:41:02.853331+0000","last_undegraded":"2024-03-24T11:41:02.853331+0000","last_fullsized":"2024-03-24T11:41:02.853331+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-03-24T11:38:51.903639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:51.903639+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":870,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853016+0000","last_change":"2024-03-24T11:38:55.092421+0000","last_active":"2024-03-24T11:41:02.853016+0000","last_peered":"2024-03-24T11:41:02.853016+0000","last_clean":"2024-03-24T11:41:02.853016+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:41:02.853016+0000","last_undegraded":"2024-03-24T11:41:02.853016+0000","last_fullsized":"2024-03-24T11:41:02.853016+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:55.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:55.092309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":1046,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852327+0000","last_change":"2024-03-24T11:41:01.109923+0000","last_active":"2024-03-24T11:41:02.852327+0000","last_peered":"2024-03-24T11:41:02.852327+0000","last_clean":"2024-03-24T11:41:02.852327+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:41:02.852327+0000","last_undegraded":"2024-03-24T11:41:02.852327+0000","last_fullsized":"2024-03-24T11:41:02.852327+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:41:01.109819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:41:01.109819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:26.826753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":184,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.384438+0000","last_change":"2024-03-24T11:39:28.567930+0000","last_active":"2024-03-24T11:41:05.384438+0000","last_peered":"2024-03-24T11:41:05.384438+0000","last_clean":"2024-03-24T11:41:05.384438+0000","last_became_active":"2024-03-24T11:39:28.567528+0000","last_became_peered":"2024-03-24T11:39:28.567528+0000","last_unstale":"2024-03-24T11:41:05.384438+0000","last_undegraded":"2024-03-24T11:41:05.384438+0000","last_fullsized":"2024-03-24T11:41:05.384438+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:41.082984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:37:41.082984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"421'42","reported_seq":949,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.384131+0000","last_change":"2024-03-24T11:39:28.610844+0000","last_active":"2024-03-24T11:41:05.384131+0000","last_peered":"2024-03-24T11:41:05.384131+0000","last_clean":"2024-03-24T11:41:05.384131+0000","last_became_active":"2024-03-24T11:39:28.588159+0000","last_became_peered":"2024-03-24T11:39:28.588159+0000","last_unstale":"2024-03-24T11:41:05.384131+0000","last_undegraded":"2024-03-24T11:41:05.384131+0000","last_fullsized":"2024-03-24T11:41:05.384131+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'30","last_scrub_stamp":"2024-03-24T11:37:32.916579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:32.916579+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":874,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386523+0000","last_change":"2024-03-24T11:39:28.568155+0000","last_active":"2024-03-24T11:41:05.386523+0000","last_peered":"2024-03-24T11:41:05.386523+0000","last_clean":"2024-03-24T11:41:05.386523+0000","last_became_active":"2024-03-24T11:39:28.566304+0000","last_became_peered":"2024-03-24T11:39:28.566304+0000","last_unstale":"2024-03-24T11:41:05.386523+0000","last_undegraded":"2024-03-24T11:41:05.386523+0000","last_fullsized":"2024-03-24T11:41:05.386523+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:57.213332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.213332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":177,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386044+0000","last_change":"2024-03-24T11:40:13.056533+0000","last_active":"2024-03-24T11:41:05.386044+0000","last_peered":"2024-03-24T11:41:05.386044+0000","last_clean":"2024-03-24T11:41:05.386044+0000","last_became_active":"2024-03-24T11:36:58.574742+0000","last_became_peered":"2024-03-24T11:36:58.574742+0000","last_unstale":"2024-03-24T11:41:05.386044+0000","last_undegraded":"2024-03-24T11:41:05.386044+0000","last_fullsized":"2024-03-24T11:41:05.386044+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:13.056204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:40:13.056204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.868634+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":498,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.306401+0000","last_change":"2024-03-24T11:41:02.315090+0000","last_active":"2024-03-24T11:41:03.306401+0000","last_peered":"2024-03-24T11:41:03.306401+0000","last_clean":"2024-03-24T11:41:03.306401+0000","last_became_active":"2024-03-24T11:37:01.307977+0000","last_became_peered":"2024-03-24T11:37:01.307977+0000","last_unstale":"2024-03-24T11:41:03.306401+0000","last_undegraded":"2024-03-24T11:41:03.306401+0000","last_fullsized":"2024-03-24T11:41:03.306401+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:41:02.315008+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_clean_scrub_stamp":"2024-03-24T11:41:02.315008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:17.187965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"388'9","reported_seq":1138,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.384266+0000","last_change":"2024-03-24T11:36:58.931284+0000","last_active":"2024-03-24T11:41:05.384266+0000","last_peered":"2024-03-24T11:41:05.384266+0000","last_clean":"2024-03-24T11:41:05.384266+0000","last_became_active":"2024-03-24T11:36:58.591546+0000","last_became_peered":"2024-03-24T11:36:58.591546+0000","last_unstale":"2024-03-24T11:41:05.384266+0000","last_undegraded":"2024-03-24T11:41:05.384266+0000","last_fullsized":"2024-03-24T11:41:05.384266+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:58.931185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:58.931185+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'42","reported_seq":597,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852369+0000","last_change":"2024-03-24T11:39:12.053807+0000","last_active":"2024-03-24T11:41:02.852369+0000","last_peered":"2024-03-24T11:41:02.852369+0000","last_clean":"2024-03-24T11:41:02.852369+0000","last_became_active":"2024-03-24T11:36:58.573987+0000","last_became_peered":"2024-03-24T11:36:58.573987+0000","last_unstale":"2024-03-24T11:41:02.852369+0000","last_undegraded":"2024-03-24T11:41:02.852369+0000","last_fullsized":"2024-03-24T11:41:02.852369+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'42","last_scrub_stamp":"2024-03-24T11:39:12.053617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:12.053617+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":920,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655626+0000","last_change":"2024-03-24T11:40:56.185901+0000","last_active":"2024-03-24T11:41:02.655626+0000","last_peered":"2024-03-24T11:41:02.655626+0000","last_clean":"2024-03-24T11:41:02.655626+0000","last_became_active":"2024-03-24T11:39:28.576255+0000","last_became_peered":"2024-03-24T11:39:28.576255+0000","last_unstale":"2024-03-24T11:41:02.655626+0000","last_undegraded":"2024-03-24T11:41:02.655626+0000","last_fullsized":"2024-03-24T11:41:02.655626+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:40:56.185811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:56.185811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:00.483005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":517,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260945+0000","last_change":"2024-03-24T11:40:57.351187+0000","last_active":"2024-03-24T11:41:03.260945+0000","last_peered":"2024-03-24T11:41:03.260945+0000","last_clean":"2024-03-24T11:41:03.260945+0000","last_became_active":"2024-03-24T11:39:28.572211+0000","last_became_peered":"2024-03-24T11:39:28.572211+0000","last_unstale":"2024-03-24T11:41:03.260945+0000","last_undegraded":"2024-03-24T11:41:03.260945+0000","last_fullsized":"2024-03-24T11:41:03.260945+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:57.351070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:57.351070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:14.159054+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1073,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386167+0000","last_change":"2024-03-24T11:39:31.390790+0000","last_active":"2024-03-24T11:41:05.386167+0000","last_peered":"2024-03-24T11:41:05.386167+0000","last_clean":"2024-03-24T11:41:05.386167+0000","last_became_active":"2024-03-24T11:39:31.390386+0000","last_became_peered":"2024-03-24T11:39:31.390386+0000","last_unstale":"2024-03-24T11:41:05.386167+0000","last_undegraded":"2024-03-24T11:41:05.386167+0000","last_fullsized":"2024-03-24T11:41:05.386167+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:42.045652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:42.045652+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":1166,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.262826+0000","last_change":"2024-03-24T11:41:03.262913+0000","last_active":"2024-03-24T11:41:03.262826+0000","last_peered":"2024-03-24T11:41:03.262826+0000","last_clean":"2024-03-24T11:41:03.262826+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:41:03.262826+0000","last_undegraded":"2024-03-24T11:41:03.262826+0000","last_fullsized":"2024-03-24T11:41:03.262826+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:41:03.262723+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:41:03.262723+0000","last_clean_scrub_stamp":"2024-03-24T11:41:03.262723+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:26.596003+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'30","reported_seq":958,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327032+0000","last_change":"2024-03-24T11:38:53.578501+0000","last_active":"2024-03-24T11:41:03.327032+0000","last_peered":"2024-03-24T11:41:03.327032+0000","last_clean":"2024-03-24T11:41:03.327032+0000","last_became_active":"2024-03-24T11:36:58.597641+0000","last_became_peered":"2024-03-24T11:36:58.597641+0000","last_unstale":"2024-03-24T11:41:03.327032+0000","last_undegraded":"2024-03-24T11:41:03.327032+0000","last_fullsized":"2024-03-24T11:41:03.327032+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'20","last_scrub_stamp":"2024-03-24T11:38:53.578365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.578365+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261102+0000","last_change":"2024-03-24T11:40:35.291321+0000","last_active":"2024-03-24T11:41:03.261102+0000","last_peered":"2024-03-24T11:41:03.261102+0000","last_clean":"2024-03-24T11:41:03.261102+0000","last_became_active":"2024-03-24T11:40:35.291074+0000","last_became_peered":"2024-03-24T11:40:35.291074+0000","last_unstale":"2024-03-24T11:41:03.261102+0000","last_undegraded":"2024-03-24T11:41:03.261102+0000","last_fullsized":"2024-03-24T11:41:03.261102+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:16.120029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:42.448705+0000","last_clean_scrub_stamp":"2024-03-24T11:39:16.120029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":880,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.653510+0000","last_change":"2024-03-24T11:40:57.258985+0000","last_active":"2024-03-24T11:41:02.653510+0000","last_peered":"2024-03-24T11:41:02.653510+0000","last_clean":"2024-03-24T11:41:02.653510+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:41:02.653510+0000","last_undegraded":"2024-03-24T11:41:02.653510+0000","last_fullsized":"2024-03-24T11:41:02.653510+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:40:57.258803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:57.258803+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:13.494335+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1098,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261360+0000","last_change":"2024-03-24T11:40:14.233673+0000","last_active":"2024-03-24T11:41:03.261360+0000","last_peered":"2024-03-24T11:41:03.261360+0000","last_clean":"2024-03-24T11:41:03.261360+0000","last_became_active":"2024-03-24T11:39:31.395565+0000","last_became_peered":"2024-03-24T11:39:31.395565+0000","last_unstale":"2024-03-24T11:41:03.261360+0000","last_undegraded":"2024-03-24T11:41:03.261360+0000","last_fullsized":"2024-03-24T11:41:03.261360+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:40:14.233591+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:31:42.062151+0000","last_clean_scrub_stamp":"2024-03-24T11:40:14.233591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:29.225205+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":907,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260799+0000","last_change":"2024-03-24T11:40:21.311312+0000","last_active":"2024-03-24T11:41:03.260799+0000","last_peered":"2024-03-24T11:41:03.260799+0000","last_clean":"2024-03-24T11:41:03.260799+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:41:03.260799+0000","last_undegraded":"2024-03-24T11:41:03.260799+0000","last_fullsized":"2024-03-24T11:41:03.260799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:21.311216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:37.391291+0000","last_clean_scrub_stamp":"2024-03-24T11:40:21.311216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:30.401945+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'39","reported_seq":977,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655012+0000","last_change":"2024-03-24T11:40:26.282042+0000","last_active":"2024-03-24T11:41:02.655012+0000","last_peered":"2024-03-24T11:41:02.655012+0000","last_clean":"2024-03-24T11:41:02.655012+0000","last_became_active":"2024-03-24T11:39:28.550731+0000","last_became_peered":"2024-03-24T11:39:28.550731+0000","last_unstale":"2024-03-24T11:41:02.655012+0000","last_undegraded":"2024-03-24T11:41:02.655012+0000","last_fullsized":"2024-03-24T11:41:02.655012+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:40:26.281902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:26.281902+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.052335+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":899,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327110+0000","last_change":"2024-03-24T11:37:49.656241+0000","last_active":"2024-03-24T11:41:03.327110+0000","last_peered":"2024-03-24T11:41:03.327110+0000","last_clean":"2024-03-24T11:41:03.327110+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:41:03.327110+0000","last_undegraded":"2024-03-24T11:41:03.327110+0000","last_fullsized":"2024-03-24T11:41:03.327110+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:49.656104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:49.656104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":887,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326705+0000","last_change":"2024-03-24T11:41:01.787698+0000","last_active":"2024-03-24T11:41:03.326705+0000","last_peered":"2024-03-24T11:41:03.326705+0000","last_clean":"2024-03-24T11:41:03.326705+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:41:03.326705+0000","last_undegraded":"2024-03-24T11:41:03.326705+0000","last_fullsized":"2024-03-24T11:41:03.326705+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:01.787579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:41:01.787579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:30.857698+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1083,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261265+0000","last_change":"2024-03-24T11:39:14.270225+0000","last_active":"2024-03-24T11:41:03.261265+0000","last_peered":"2024-03-24T11:41:03.261265+0000","last_clean":"2024-03-24T11:41:03.261265+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:41:03.261265+0000","last_undegraded":"2024-03-24T11:41:03.261265+0000","last_fullsized":"2024-03-24T11:41:03.261265+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_clean_scrub_stamp":"2024-03-24T11:39:14.270084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":883,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853614+0000","last_change":"2024-03-24T11:39:14.156792+0000","last_active":"2024-03-24T11:41:02.853614+0000","last_peered":"2024-03-24T11:41:02.853614+0000","last_clean":"2024-03-24T11:41:02.853614+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:41:02.853614+0000","last_undegraded":"2024-03-24T11:41:02.853614+0000","last_fullsized":"2024-03-24T11:41:02.853614+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:14.156684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:39:14.156684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'36","reported_seq":450,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853218+0000","last_change":"2024-03-24T11:39:01.923217+0000","last_active":"2024-03-24T11:41:02.853218+0000","last_peered":"2024-03-24T11:41:02.853218+0000","last_clean":"2024-03-24T11:41:02.853218+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:41:02.853218+0000","last_undegraded":"2024-03-24T11:41:02.853218+0000","last_fullsized":"2024-03-24T11:41:02.853218+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'28","last_scrub_stamp":"2024-03-24T11:39:01.923083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.923083+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1182,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326651+0000","last_change":"2024-03-24T11:40:25.760465+0000","last_active":"2024-03-24T11:41:03.326651+0000","last_peered":"2024-03-24T11:41:03.326651+0000","last_clean":"2024-03-24T11:41:03.326651+0000","last_became_active":"2024-03-24T11:39:28.571981+0000","last_became_peered":"2024-03-24T11:39:28.571981+0000","last_unstale":"2024-03-24T11:41:03.326651+0000","last_undegraded":"2024-03-24T11:41:03.326651+0000","last_fullsized":"2024-03-24T11:41:03.326651+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:40:25.760218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:40:25.760218+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.257271+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":628,"num_read_kb":623,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":900,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655037+0000","last_change":"2024-03-24T11:39:06.423843+0000","last_active":"2024-03-24T11:41:02.655037+0000","last_peered":"2024-03-24T11:41:02.655037+0000","last_clean":"2024-03-24T11:41:02.655037+0000","last_became_active":"2024-03-24T11:36:58.572807+0000","last_became_peered":"2024-03-24T11:36:58.572807+0000","last_unstale":"2024-03-24T11:41:02.655037+0000","last_undegraded":"2024-03-24T11:41:02.655037+0000","last_fullsized":"2024-03-24T11:41:02.655037+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:06.423513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:39:06.423513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":887,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326972+0000","last_change":"2024-03-24T11:38:05.656269+0000","last_active":"2024-03-24T11:41:03.326972+0000","last_peered":"2024-03-24T11:41:03.326972+0000","last_clean":"2024-03-24T11:41:03.326972+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:41:03.326972+0000","last_undegraded":"2024-03-24T11:41:03.326972+0000","last_fullsized":"2024-03-24T11:41:03.326972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:05.656146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.778835+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.656146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1094,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326987+0000","last_change":"2024-03-24T11:40:15.791385+0000","last_active":"2024-03-24T11:41:03.326987+0000","last_peered":"2024-03-24T11:41:03.326987+0000","last_clean":"2024-03-24T11:41:03.326987+0000","last_became_active":"2024-03-24T11:37:01.307925+0000","last_became_peered":"2024-03-24T11:37:01.307925+0000","last_unstale":"2024-03-24T11:41:03.326987+0000","last_undegraded":"2024-03-24T11:41:03.326987+0000","last_fullsized":"2024-03-24T11:41:03.326987+0000","mapping_epoch":390,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:40:15.790434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.790434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:15.988035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":517,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.306397+0000","last_change":"2024-03-24T11:40:11.295320+0000","last_active":"2024-03-24T11:41:03.306397+0000","last_peered":"2024-03-24T11:41:03.306397+0000","last_clean":"2024-03-24T11:41:03.306397+0000","last_became_active":"2024-03-24T11:39:28.569858+0000","last_became_peered":"2024-03-24T11:39:28.569858+0000","last_unstale":"2024-03-24T11:41:03.306397+0000","last_undegraded":"2024-03-24T11:41:03.306397+0000","last_fullsized":"2024-03-24T11:41:03.306397+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:11.295201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.295201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:41.218881+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"429'50","reported_seq":1291,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261347+0000","last_change":"2024-03-24T11:40:55.208483+0000","last_active":"2024-03-24T11:41:03.261347+0000","last_peered":"2024-03-24T11:41:03.261347+0000","last_clean":"2024-03-24T11:41:03.261347+0000","last_became_active":"2024-03-24T11:39:28.564381+0000","last_became_peered":"2024-03-24T11:39:28.564381+0000","last_unstale":"2024-03-24T11:41:03.261347+0000","last_undegraded":"2024-03-24T11:41:03.261347+0000","last_fullsized":"2024-03-24T11:41:03.261347+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'50","last_scrub_stamp":"2024-03-24T11:40:55.208344+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:40:55.208344+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:18.277885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":880,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.385801+0000","last_change":"2024-03-24T11:39:28.585112+0000","last_active":"2024-03-24T11:41:05.385801+0000","last_peered":"2024-03-24T11:41:05.385801+0000","last_clean":"2024-03-24T11:41:05.385801+0000","last_became_active":"2024-03-24T11:39:28.584724+0000","last_became_peered":"2024-03-24T11:39:28.584724+0000","last_unstale":"2024-03-24T11:41:05.385801+0000","last_undegraded":"2024-03-24T11:41:05.385801+0000","last_fullsized":"2024-03-24T11:41:05.385801+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:35.978638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:35.978638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":846,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.654109+0000","last_change":"2024-03-24T11:39:31.397589+0000","last_active":"2024-03-24T11:41:02.654109+0000","last_peered":"2024-03-24T11:41:02.654109+0000","last_clean":"2024-03-24T11:41:02.654109+0000","last_became_active":"2024-03-24T11:39:31.397237+0000","last_became_peered":"2024-03-24T11:39:31.397237+0000","last_unstale":"2024-03-24T11:41:02.654109+0000","last_undegraded":"2024-03-24T11:41:02.654109+0000","last_fullsized":"2024-03-24T11:41:02.654109+0000","mapping_epoch":431,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:39:03.394244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:39:03.394244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":893,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853441+0000","last_change":"2024-03-24T11:38:02.291361+0000","last_active":"2024-03-24T11:41:02.853441+0000","last_peered":"2024-03-24T11:41:02.853441+0000","last_clean":"2024-03-24T11:41:02.853441+0000","last_became_active":"2024-03-24T11:36:58.586913+0000","last_became_peered":"2024-03-24T11:36:58.586913+0000","last_unstale":"2024-03-24T11:41:02.853441+0000","last_undegraded":"2024-03-24T11:41:02.853441+0000","last_fullsized":"2024-03-24T11:41:02.853441+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:02.291240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:38:02.291240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":921,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.306369+0000","last_change":"2024-03-24T11:41:01.268470+0000","last_active":"2024-03-24T11:41:03.306369+0000","last_peered":"2024-03-24T11:41:03.306369+0000","last_clean":"2024-03-24T11:41:03.306369+0000","last_became_active":"2024-03-24T11:39:28.574632+0000","last_became_peered":"2024-03-24T11:39:28.574632+0000","last_unstale":"2024-03-24T11:41:03.306369+0000","last_undegraded":"2024-03-24T11:41:03.306369+0000","last_fullsized":"2024-03-24T11:41:03.306369+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:01.268369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:41:01.268369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:08.430094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"449'26","reported_seq":930,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.305503+0000","last_change":"2024-03-24T11:40:22.408424+0000","last_active":"2024-03-24T11:41:03.305503+0000","last_peered":"2024-03-24T11:41:03.305503+0000","last_clean":"2024-03-24T11:41:03.305503+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:41:03.305503+0000","last_undegraded":"2024-03-24T11:41:03.305503+0000","last_fullsized":"2024-03-24T11:41:03.305503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'26","last_scrub_stamp":"2024-03-24T11:40:22.408163+0000","last_deep_scrub":"449'26","last_deep_scrub_stamp":"2024-03-24T11:40:22.408163+0000","last_clean_scrub_stamp":"2024-03-24T11:40:22.408163+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:42.483835+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":924,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.305721+0000","last_change":"2024-03-24T11:40:15.313269+0000","last_active":"2024-03-24T11:41:03.305721+0000","last_peered":"2024-03-24T11:41:03.305721+0000","last_clean":"2024-03-24T11:41:03.305721+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:41:03.305721+0000","last_undegraded":"2024-03-24T11:41:03.305721+0000","last_fullsized":"2024-03-24T11:41:03.305721+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:15.313154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:23:50.832319+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.313154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:20.321870+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":900,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326768+0000","last_change":"2024-03-24T11:37:53.683206+0000","last_active":"2024-03-24T11:41:03.326768+0000","last_peered":"2024-03-24T11:41:03.326768+0000","last_clean":"2024-03-24T11:41:03.326768+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:41:03.326768+0000","last_undegraded":"2024-03-24T11:41:03.326768+0000","last_fullsized":"2024-03-24T11:41:03.326768+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:53.683090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:53.683090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":517,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.374265+0000","last_change":"2024-03-24T11:41:03.374265+0000","last_active":"2024-03-24T11:41:03.374265+0000","last_peered":"2024-03-24T11:41:03.374265+0000","last_clean":"2024-03-24T11:41:03.374265+0000","last_became_active":"2024-03-24T11:39:28.561579+0000","last_became_peered":"2024-03-24T11:39:28.561579+0000","last_unstale":"2024-03-24T11:41:03.374265+0000","last_undegraded":"2024-03-24T11:41:03.374265+0000","last_fullsized":"2024-03-24T11:41:03.374265+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:03.374157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:41:03.374157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:18.212939+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":871,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.654457+0000","last_change":"2024-03-24T11:40:13.361900+0000","last_active":"2024-03-24T11:41:02.654457+0000","last_peered":"2024-03-24T11:41:02.654457+0000","last_clean":"2024-03-24T11:41:02.654457+0000","last_became_active":"2024-03-24T11:39:30.375769+0000","last_became_peered":"2024-03-24T11:39:30.375769+0000","last_unstale":"2024-03-24T11:41:02.654457+0000","last_undegraded":"2024-03-24T11:41:02.654457+0000","last_fullsized":"2024-03-24T11:41:02.654457+0000","mapping_epoch":430,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:40:13.361480+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:40:13.361480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:25.011763+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":889,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.654385+0000","last_change":"2024-03-24T11:40:18.336459+0000","last_active":"2024-03-24T11:41:02.654385+0000","last_peered":"2024-03-24T11:41:02.654385+0000","last_clean":"2024-03-24T11:41:02.654385+0000","last_became_active":"2024-03-24T11:36:58.594154+0000","last_became_peered":"2024-03-24T11:36:58.594154+0000","last_unstale":"2024-03-24T11:41:02.654385+0000","last_undegraded":"2024-03-24T11:41:02.654385+0000","last_fullsized":"2024-03-24T11:41:02.654385+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:18.336351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:18.336351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:26.769793+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"421'53","reported_seq":1273,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327684+0000","last_change":"2024-03-24T11:41:00.727702+0000","last_active":"2024-03-24T11:41:03.327684+0000","last_peered":"2024-03-24T11:41:03.327684+0000","last_clean":"2024-03-24T11:41:03.327684+0000","last_became_active":"2024-03-24T11:37:33.888447+0000","last_became_peered":"2024-03-24T11:37:33.888447+0000","last_unstale":"2024-03-24T11:41:03.327684+0000","last_undegraded":"2024-03-24T11:41:03.327684+0000","last_fullsized":"2024-03-24T11:41:03.327684+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'53","last_scrub_stamp":"2024-03-24T11:41:00.727591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:00.727591+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:21.411187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":183,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.654924+0000","last_change":"2024-03-24T11:40:22.347875+0000","last_active":"2024-03-24T11:41:02.654924+0000","last_peered":"2024-03-24T11:41:02.654924+0000","last_clean":"2024-03-24T11:41:02.654924+0000","last_became_active":"2024-03-24T11:36:58.591467+0000","last_became_peered":"2024-03-24T11:36:58.591467+0000","last_unstale":"2024-03-24T11:41:02.654924+0000","last_undegraded":"2024-03-24T11:41:02.654924+0000","last_fullsized":"2024-03-24T11:41:02.654924+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:22.347618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_clean_scrub_stamp":"2024-03-24T11:40:22.347618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:43.963473+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":902,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852468+0000","last_change":"2024-03-24T11:39:05.008974+0000","last_active":"2024-03-24T11:41:02.852468+0000","last_peered":"2024-03-24T11:41:02.852468+0000","last_clean":"2024-03-24T11:41:02.852468+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:41:02.852468+0000","last_undegraded":"2024-03-24T11:41:02.852468+0000","last_fullsized":"2024-03-24T11:41:02.852468+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:39:05.008874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:39:05.008874+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":868,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386605+0000","last_change":"2024-03-24T11:39:28.580748+0000","last_active":"2024-03-24T11:41:05.386605+0000","last_peered":"2024-03-24T11:41:05.386605+0000","last_clean":"2024-03-24T11:41:05.386605+0000","last_became_active":"2024-03-24T11:39:28.576107+0000","last_became_peered":"2024-03-24T11:39:28.576107+0000","last_unstale":"2024-03-24T11:41:05.386605+0000","last_undegraded":"2024-03-24T11:41:05.386605+0000","last_fullsized":"2024-03-24T11:41:05.386605+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:02.217239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:02.217239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1069,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852362+0000","last_change":"2024-03-24T11:37:55.105164+0000","last_active":"2024-03-24T11:41:02.852362+0000","last_peered":"2024-03-24T11:41:02.852362+0000","last_clean":"2024-03-24T11:41:02.852362+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:41:02.852362+0000","last_undegraded":"2024-03-24T11:41:02.852362+0000","last_fullsized":"2024-03-24T11:41:02.852362+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:37:55.105007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:37:55.105007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655248+0000","last_change":"2024-03-24T11:39:12.419766+0000","last_active":"2024-03-24T11:41:02.655248+0000","last_peered":"2024-03-24T11:41:02.655248+0000","last_clean":"2024-03-24T11:41:02.655248+0000","last_became_active":"2024-03-24T11:36:58.590879+0000","last_became_peered":"2024-03-24T11:36:58.590879+0000","last_unstale":"2024-03-24T11:41:02.655248+0000","last_undegraded":"2024-03-24T11:41:02.655248+0000","last_fullsized":"2024-03-24T11:41:02.655248+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:39:12.418838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:12.418838+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"66'1","reported_seq":896,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852891+0000","last_change":"2024-03-24T11:38:05.022910+0000","last_active":"2024-03-24T11:41:02.852891+0000","last_peered":"2024-03-24T11:41:02.852891+0000","last_clean":"2024-03-24T11:41:02.852891+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:41:02.852891+0000","last_undegraded":"2024-03-24T11:41:02.852891+0000","last_fullsized":"2024-03-24T11:41:02.852891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:38:05.022785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:05.022785+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":179,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.385919+0000","last_change":"2024-03-24T11:39:01.186035+0000","last_active":"2024-03-24T11:41:05.385919+0000","last_peered":"2024-03-24T11:41:05.385919+0000","last_clean":"2024-03-24T11:41:05.385919+0000","last_became_active":"2024-03-24T11:36:58.576200+0000","last_became_peered":"2024-03-24T11:36:58.576200+0000","last_unstale":"2024-03-24T11:41:05.385919+0000","last_undegraded":"2024-03-24T11:41:05.385919+0000","last_fullsized":"2024-03-24T11:41:05.385919+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:01.185721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:01.185721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":885,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655375+0000","last_change":"2024-03-24T11:40:16.413527+0000","last_active":"2024-03-24T11:41:02.655375+0000","last_peered":"2024-03-24T11:41:02.655375+0000","last_clean":"2024-03-24T11:41:02.655375+0000","last_became_active":"2024-03-24T11:39:28.567784+0000","last_became_peered":"2024-03-24T11:39:28.567784+0000","last_unstale":"2024-03-24T11:41:02.655375+0000","last_undegraded":"2024-03-24T11:41:02.655375+0000","last_fullsized":"2024-03-24T11:41:02.655375+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:16.413401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:16.413401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:41.494869+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":1017,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260750+0000","last_change":"2024-03-24T11:40:11.195573+0000","last_active":"2024-03-24T11:41:03.260750+0000","last_peered":"2024-03-24T11:41:03.260750+0000","last_clean":"2024-03-24T11:41:03.260750+0000","last_became_active":"2024-03-24T11:39:28.547142+0000","last_became_peered":"2024-03-24T11:39:28.547142+0000","last_unstale":"2024-03-24T11:41:03.260750+0000","last_undegraded":"2024-03-24T11:41:03.260750+0000","last_fullsized":"2024-03-24T11:41:03.260750+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'60","last_scrub_stamp":"2024-03-24T11:40:11.195421+0000","last_deep_scrub":"88'28","last_deep_scrub_stamp":"2024-03-24T11:20:23.477264+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.195421+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:17.973047+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":506,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853210+0000","last_change":"2024-03-24T11:40:34.982216+0000","last_active":"2024-03-24T11:41:02.853210+0000","last_peered":"2024-03-24T11:41:02.853210+0000","last_clean":"2024-03-24T11:41:02.853210+0000","last_became_active":"2024-03-24T11:40:34.981873+0000","last_became_peered":"2024-03-24T11:40:34.981873+0000","last_unstale":"2024-03-24T11:41:02.853210+0000","last_undegraded":"2024-03-24T11:41:02.853210+0000","last_fullsized":"2024-03-24T11:41:02.853210+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:46.876805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:46.876805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":914,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.305920+0000","last_change":"2024-03-24T11:40:58.239024+0000","last_active":"2024-03-24T11:41:03.305920+0000","last_peered":"2024-03-24T11:41:03.305920+0000","last_clean":"2024-03-24T11:41:03.305920+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:41:03.305920+0000","last_undegraded":"2024-03-24T11:41:03.305920+0000","last_fullsized":"2024-03-24T11:41:03.305920+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:58.238935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:40:58.238935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:26.797965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":203,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.383731+0000","last_change":"2024-03-24T11:39:28.587339+0000","last_active":"2024-03-24T11:41:05.383731+0000","last_peered":"2024-03-24T11:41:05.383731+0000","last_clean":"2024-03-24T11:41:05.383731+0000","last_became_active":"2024-03-24T11:39:28.575619+0000","last_became_peered":"2024-03-24T11:39:28.575619+0000","last_unstale":"2024-03-24T11:41:05.383731+0000","last_undegraded":"2024-03-24T11:41:05.383731+0000","last_fullsized":"2024-03-24T11:41:05.383731+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.123323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":202,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.654425+0000","last_change":"2024-03-24T11:40:53.244059+0000","last_active":"2024-03-24T11:41:02.654425+0000","last_peered":"2024-03-24T11:41:02.654425+0000","last_clean":"2024-03-24T11:41:02.654425+0000","last_became_active":"2024-03-24T11:36:42.470376+0000","last_became_peered":"2024-03-24T11:36:42.470376+0000","last_unstale":"2024-03-24T11:41:02.654425+0000","last_undegraded":"2024-03-24T11:41:02.654425+0000","last_fullsized":"2024-03-24T11:41:02.654425+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:53.243956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:53.243956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:03.661987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"421'24","reported_seq":571,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326578+0000","last_change":"2024-03-24T11:38:52.571057+0000","last_active":"2024-03-24T11:41:03.326578+0000","last_peered":"2024-03-24T11:41:03.326578+0000","last_clean":"2024-03-24T11:41:03.326578+0000","last_became_active":"2024-03-24T11:36:58.593181+0000","last_became_peered":"2024-03-24T11:36:58.593181+0000","last_unstale":"2024-03-24T11:41:03.326578+0000","last_undegraded":"2024-03-24T11:41:03.326578+0000","last_fullsized":"2024-03-24T11:41:03.326578+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:38:52.570919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:52.570919+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":191,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327683+0000","last_change":"2024-03-24T11:40:59.743865+0000","last_active":"2024-03-24T11:41:03.327683+0000","last_peered":"2024-03-24T11:41:03.327683+0000","last_clean":"2024-03-24T11:41:03.327683+0000","last_became_active":"2024-03-24T11:39:28.578963+0000","last_became_peered":"2024-03-24T11:39:28.578963+0000","last_unstale":"2024-03-24T11:41:03.327683+0000","last_undegraded":"2024-03-24T11:41:03.327683+0000","last_fullsized":"2024-03-24T11:41:03.327683+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:59.743794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:40:59.743794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:27.294371+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":861,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386236+0000","last_change":"2024-03-24T11:40:26.396360+0000","last_active":"2024-03-24T11:41:05.386236+0000","last_peered":"2024-03-24T11:41:05.386236+0000","last_clean":"2024-03-24T11:41:05.386236+0000","last_became_active":"2024-03-24T11:39:28.581732+0000","last_became_peered":"2024-03-24T11:39:28.581732+0000","last_unstale":"2024-03-24T11:41:05.386236+0000","last_undegraded":"2024-03-24T11:41:05.386236+0000","last_fullsized":"2024-03-24T11:41:05.386236+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:26.396256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:26.396256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:39.252686+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"429'12","reported_seq":682,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260483+0000","last_change":"2024-03-24T11:40:54.205084+0000","last_active":"2024-03-24T11:41:03.260483+0000","last_peered":"2024-03-24T11:41:03.260483+0000","last_clean":"2024-03-24T11:41:03.260483+0000","last_became_active":"2024-03-24T11:39:28.577500+0000","last_became_peered":"2024-03-24T11:39:28.577500+0000","last_unstale":"2024-03-24T11:41:03.260483+0000","last_undegraded":"2024-03-24T11:41:03.260483+0000","last_fullsized":"2024-03-24T11:41:03.260483+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'12","last_scrub_stamp":"2024-03-24T11:40:54.204976+0000","last_deep_scrub":"357'10","last_deep_scrub_stamp":"2024-03-24T11:35:53.386928+0000","last_clean_scrub_stamp":"2024-03-24T11:40:54.204976+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:06.153466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"421'46","reported_seq":985,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852077+0000","last_change":"2024-03-24T11:41:00.077444+0000","last_active":"2024-03-24T11:41:02.852077+0000","last_peered":"2024-03-24T11:41:02.852077+0000","last_clean":"2024-03-24T11:41:02.852077+0000","last_became_active":"2024-03-24T11:39:28.563427+0000","last_became_peered":"2024-03-24T11:39:28.563427+0000","last_unstale":"2024-03-24T11:41:02.852077+0000","last_undegraded":"2024-03-24T11:41:02.852077+0000","last_fullsized":"2024-03-24T11:41:02.852077+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-24T11:41:00.077316+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:41:00.077316+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:03.606347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":403,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655863+0000","last_change":"2024-03-24T11:40:21.348781+0000","last_active":"2024-03-24T11:41:02.655863+0000","last_peered":"2024-03-24T11:41:02.655863+0000","last_clean":"2024-03-24T11:41:02.655863+0000","last_became_active":"2024-03-24T11:39:28.582617+0000","last_became_peered":"2024-03-24T11:39:28.582617+0000","last_unstale":"2024-03-24T11:41:02.655863+0000","last_undegraded":"2024-03-24T11:41:02.655863+0000","last_fullsized":"2024-03-24T11:41:02.655863+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:40:21.348646+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:40:21.348646+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:35.106715+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":879,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852696+0000","last_change":"2024-03-24T11:38:04.034506+0000","last_active":"2024-03-24T11:41:02.852696+0000","last_peered":"2024-03-24T11:41:02.852696+0000","last_clean":"2024-03-24T11:41:02.852696+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:41:02.852696+0000","last_undegraded":"2024-03-24T11:41:02.852696+0000","last_fullsized":"2024-03-24T11:41:02.852696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:04.034411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:04.034411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":907,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386720+0000","last_change":"2024-03-24T11:41:02.220240+0000","last_active":"2024-03-24T11:41:05.386720+0000","last_peered":"2024-03-24T11:41:05.386720+0000","last_clean":"2024-03-24T11:41:05.386720+0000","last_became_active":"2024-03-24T11:39:28.583847+0000","last_became_peered":"2024-03-24T11:39:28.583847+0000","last_unstale":"2024-03-24T11:41:05.386720+0000","last_undegraded":"2024-03-24T11:41:05.386720+0000","last_fullsized":"2024-03-24T11:41:05.386720+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:41:02.220130+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:41:02.220130+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:07.830827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":914,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260137+0000","last_change":"2024-03-24T11:40:26.287970+0000","last_active":"2024-03-24T11:41:03.260137+0000","last_peered":"2024-03-24T11:41:03.260137+0000","last_clean":"2024-03-24T11:41:03.260137+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:41:03.260137+0000","last_undegraded":"2024-03-24T11:41:03.260137+0000","last_fullsized":"2024-03-24T11:41:03.260137+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:26.287872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:26.287872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:41.149626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":882,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853228+0000","last_change":"2024-03-24T11:40:22.248773+0000","last_active":"2024-03-24T11:41:02.853228+0000","last_peered":"2024-03-24T11:41:02.853228+0000","last_clean":"2024-03-24T11:41:02.853228+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:41:02.853228+0000","last_undegraded":"2024-03-24T11:41:02.853228+0000","last_fullsized":"2024-03-24T11:41:02.853228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:22.248686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:22.248686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:45.429437+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":522,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261442+0000","last_change":"2024-03-24T11:40:24.442702+0000","last_active":"2024-03-24T11:41:03.261442+0000","last_peered":"2024-03-24T11:41:03.261442+0000","last_clean":"2024-03-24T11:41:03.261442+0000","last_became_active":"2024-03-24T11:39:28.547484+0000","last_became_peered":"2024-03-24T11:39:28.547484+0000","last_unstale":"2024-03-24T11:41:03.261442+0000","last_undegraded":"2024-03-24T11:41:03.261442+0000","last_fullsized":"2024-03-24T11:41:03.261442+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:24.442599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:24.442599+0000","last_clean_scrub_stamp":"2024-03-24T11:40:24.442599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:45.397849+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"421'25","reported_seq":940,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655726+0000","last_change":"2024-03-24T11:39:28.590958+0000","last_active":"2024-03-24T11:41:02.655726+0000","last_peered":"2024-03-24T11:41:02.655726+0000","last_clean":"2024-03-24T11:41:02.655726+0000","last_became_active":"2024-03-24T11:39:28.564216+0000","last_became_peered":"2024-03-24T11:39:28.564216+0000","last_unstale":"2024-03-24T11:41:02.655726+0000","last_undegraded":"2024-03-24T11:41:02.655726+0000","last_fullsized":"2024-03-24T11:41:02.655726+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'19","last_scrub_stamp":"2024-03-24T11:38:57.280974+0000","last_deep_scrub":"86'13","last_deep_scrub_stamp":"2024-03-24T11:19:58.001296+0000","last_clean_scrub_stamp":"2024-03-24T11:38:57.280974+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":191,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260241+0000","last_change":"2024-03-24T11:40:12.210986+0000","last_active":"2024-03-24T11:41:03.260241+0000","last_peered":"2024-03-24T11:41:03.260241+0000","last_clean":"2024-03-24T11:41:03.260241+0000","last_became_active":"2024-03-24T11:39:28.577340+0000","last_became_peered":"2024-03-24T11:39:28.577340+0000","last_unstale":"2024-03-24T11:41:03.260241+0000","last_undegraded":"2024-03-24T11:41:03.260241+0000","last_fullsized":"2024-03-24T11:41:03.260241+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:12.210906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:40:12.210906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:12.399391+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":865,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386684+0000","last_change":"2024-03-24T11:40:20.264676+0000","last_active":"2024-03-24T11:41:05.386684+0000","last_peered":"2024-03-24T11:41:05.386684+0000","last_clean":"2024-03-24T11:41:05.386684+0000","last_became_active":"2024-03-24T11:39:28.581534+0000","last_became_peered":"2024-03-24T11:39:28.581534+0000","last_unstale":"2024-03-24T11:41:05.386684+0000","last_undegraded":"2024-03-24T11:41:05.386684+0000","last_fullsized":"2024-03-24T11:41:05.386684+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:20.263202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:40:20.263202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.422784+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":914,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.306286+0000","last_change":"2024-03-24T11:40:55.322395+0000","last_active":"2024-03-24T11:41:03.306286+0000","last_peered":"2024-03-24T11:41:03.306286+0000","last_clean":"2024-03-24T11:41:03.306286+0000","last_became_active":"2024-03-24T11:39:28.570115+0000","last_became_peered":"2024-03-24T11:39:28.570115+0000","last_unstale":"2024-03-24T11:41:03.306286+0000","last_undegraded":"2024-03-24T11:41:03.306286+0000","last_fullsized":"2024-03-24T11:41:03.306286+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:55.322293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:55.322293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:24.907856+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":952,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853336+0000","last_change":"2024-03-24T11:39:28.551439+0000","last_active":"2024-03-24T11:41:02.853336+0000","last_peered":"2024-03-24T11:41:02.853336+0000","last_clean":"2024-03-24T11:41:02.853336+0000","last_became_active":"2024-03-24T11:39:28.544005+0000","last_became_peered":"2024-03-24T11:39:28.544005+0000","last_unstale":"2024-03-24T11:41:02.853336+0000","last_undegraded":"2024-03-24T11:41:02.853336+0000","last_fullsized":"2024-03-24T11:41:02.853336+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:11.883396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:11.883396+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":896,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.654277+0000","last_change":"2024-03-24T11:39:28.575492+0000","last_active":"2024-03-24T11:41:02.654277+0000","last_peered":"2024-03-24T11:41:02.654277+0000","last_clean":"2024-03-24T11:41:02.654277+0000","last_became_active":"2024-03-24T11:39:28.575151+0000","last_became_peered":"2024-03-24T11:39:28.575151+0000","last_unstale":"2024-03-24T11:41:02.654277+0000","last_undegraded":"2024-03-24T11:41:02.654277+0000","last_fullsized":"2024-03-24T11:41:02.654277+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:07.446672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:55.793540+0000","last_clean_scrub_stamp":"2024-03-24T11:39:07.446672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"429'9","reported_seq":1144,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386562+0000","last_change":"2024-03-24T11:39:28.596125+0000","last_active":"2024-03-24T11:41:05.386562+0000","last_peered":"2024-03-24T11:41:05.386562+0000","last_clean":"2024-03-24T11:41:05.386562+0000","last_became_active":"2024-03-24T11:39:28.581602+0000","last_became_peered":"2024-03-24T11:39:28.581602+0000","last_unstale":"2024-03-24T11:41:05.386562+0000","last_undegraded":"2024-03-24T11:41:05.386562+0000","last_fullsized":"2024-03-24T11:41:05.386562+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:36:54.050291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:36:54.050291+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":191,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.306426+0000","last_change":"2024-03-24T11:40:59.421078+0000","last_active":"2024-03-24T11:41:03.306426+0000","last_peered":"2024-03-24T11:41:03.306426+0000","last_clean":"2024-03-24T11:41:03.306426+0000","last_became_active":"2024-03-24T11:36:58.580967+0000","last_became_peered":"2024-03-24T11:36:58.580967+0000","last_unstale":"2024-03-24T11:41:03.306426+0000","last_undegraded":"2024-03-24T11:41:03.306426+0000","last_fullsized":"2024-03-24T11:41:03.306426+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:59.420979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:40:59.420979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:03.445580+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":926,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386459+0000","last_change":"2024-03-24T11:40:12.102298+0000","last_active":"2024-03-24T11:41:05.386459+0000","last_peered":"2024-03-24T11:41:05.386459+0000","last_clean":"2024-03-24T11:41:05.386459+0000","last_became_active":"2024-03-24T11:39:28.577653+0000","last_became_peered":"2024-03-24T11:39:28.577653+0000","last_unstale":"2024-03-24T11:41:05.386459+0000","last_undegraded":"2024-03-24T11:41:05.386459+0000","last_fullsized":"2024-03-24T11:41:05.386459+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:40:12.102156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:12.102156+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:26.823746+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":183,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.305855+0000","last_change":"2024-03-24T11:39:28.567090+0000","last_active":"2024-03-24T11:41:03.305855+0000","last_peered":"2024-03-24T11:41:03.305855+0000","last_clean":"2024-03-24T11:41:03.305855+0000","last_became_active":"2024-03-24T11:39:28.566772+0000","last_became_peered":"2024-03-24T11:39:28.566772+0000","last_unstale":"2024-03-24T11:41:03.305855+0000","last_undegraded":"2024-03-24T11:41:03.305855+0000","last_fullsized":"2024-03-24T11:41:03.305855+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.220270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.220270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":1156,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853187+0000","last_change":"2024-03-24T11:38:08.961040+0000","last_active":"2024-03-24T11:41:02.853187+0000","last_peered":"2024-03-24T11:41:02.853187+0000","last_clean":"2024-03-24T11:41:02.853187+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:41:02.853187+0000","last_undegraded":"2024-03-24T11:41:02.853187+0000","last_fullsized":"2024-03-24T11:41:02.853187+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-24T11:38:08.960909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:38:08.960909+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":876,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655525+0000","last_change":"2024-03-24T11:39:28.561627+0000","last_active":"2024-03-24T11:41:02.655525+0000","last_peered":"2024-03-24T11:41:02.655525+0000","last_clean":"2024-03-24T11:41:02.655525+0000","last_became_active":"2024-03-24T11:39:28.561292+0000","last_became_peered":"2024-03-24T11:39:28.561292+0000","last_unstale":"2024-03-24T11:41:02.655525+0000","last_undegraded":"2024-03-24T11:41:02.655525+0000","last_fullsized":"2024-03-24T11:41:02.655525+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:09.604834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:39:09.604834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'32","reported_seq":586,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260719+0000","last_change":"2024-03-24T11:39:13.314319+0000","last_active":"2024-03-24T11:41:03.260719+0000","last_peered":"2024-03-24T11:41:03.260719+0000","last_clean":"2024-03-24T11:41:03.260719+0000","last_became_active":"2024-03-24T11:36:58.586291+0000","last_became_peered":"2024-03-24T11:36:58.586291+0000","last_unstale":"2024-03-24T11:41:03.260719+0000","last_undegraded":"2024-03-24T11:41:03.260719+0000","last_fullsized":"2024-03-24T11:41:03.260719+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:39:13.314202+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-03-24T11:25:50.138485+0000","last_clean_scrub_stamp":"2024-03-24T11:39:13.314202+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":181,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260949+0000","last_change":"2024-03-24T11:39:28.564092+0000","last_active":"2024-03-24T11:41:03.260949+0000","last_peered":"2024-03-24T11:41:03.260949+0000","last_clean":"2024-03-24T11:41:03.260949+0000","last_became_active":"2024-03-24T11:39:28.563739+0000","last_became_peered":"2024-03-24T11:39:28.563739+0000","last_unstale":"2024-03-24T11:41:03.260949+0000","last_undegraded":"2024-03-24T11:41:03.260949+0000","last_fullsized":"2024-03-24T11:41:03.260949+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:59.070123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:19:56.100633+0000","last_clean_scrub_stamp":"2024-03-24T11:38:59.070123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":898,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326997+0000","last_change":"2024-03-24T11:40:56.755467+0000","last_active":"2024-03-24T11:41:03.326997+0000","last_peered":"2024-03-24T11:41:03.326997+0000","last_clean":"2024-03-24T11:41:03.326997+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:41:03.326997+0000","last_undegraded":"2024-03-24T11:41:03.326997+0000","last_fullsized":"2024-03-24T11:41:03.326997+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:56.755386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:40:56.755386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:22.939995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":855,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.384197+0000","last_change":"2024-03-24T11:37:40.087780+0000","last_active":"2024-03-24T11:41:05.384197+0000","last_peered":"2024-03-24T11:41:05.384197+0000","last_clean":"2024-03-24T11:41:05.384197+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:41:05.384197+0000","last_undegraded":"2024-03-24T11:41:05.384197+0000","last_fullsized":"2024-03-24T11:41:05.384197+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:40.086223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:37:40.086223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":928,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327374+0000","last_change":"2024-03-24T11:41:02.784482+0000","last_active":"2024-03-24T11:41:03.327374+0000","last_peered":"2024-03-24T11:41:03.327374+0000","last_clean":"2024-03-24T11:41:03.327374+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:41:03.327374+0000","last_undegraded":"2024-03-24T11:41:03.327374+0000","last_fullsized":"2024-03-24T11:41:03.327374+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:41:02.784356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:02.784356+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:12.165348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":499,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386018+0000","last_change":"2024-03-24T11:39:28.579890+0000","last_active":"2024-03-24T11:41:05.386018+0000","last_peered":"2024-03-24T11:41:05.386018+0000","last_clean":"2024-03-24T11:41:05.386018+0000","last_became_active":"2024-03-24T11:39:28.575621+0000","last_became_peered":"2024-03-24T11:39:28.575621+0000","last_unstale":"2024-03-24T11:41:05.386018+0000","last_undegraded":"2024-03-24T11:41:05.386018+0000","last_fullsized":"2024-03-24T11:41:05.386018+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:51.172813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:51.172813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":859,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386007+0000","last_change":"2024-03-24T11:39:15.574465+0000","last_active":"2024-03-24T11:41:05.386007+0000","last_peered":"2024-03-24T11:41:05.386007+0000","last_clean":"2024-03-24T11:41:05.386007+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:41:05.386007+0000","last_undegraded":"2024-03-24T11:41:05.386007+0000","last_fullsized":"2024-03-24T11:41:05.386007+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:15.574327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:15.574327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":878,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.653414+0000","last_change":"2024-03-24T11:40:09.537029+0000","last_active":"2024-03-24T11:41:02.653414+0000","last_peered":"2024-03-24T11:41:02.653414+0000","last_clean":"2024-03-24T11:41:02.653414+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:41:02.653414+0000","last_undegraded":"2024-03-24T11:41:02.653414+0000","last_fullsized":"2024-03-24T11:41:02.653414+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:09.536910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:09.536910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.356810+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":924,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.384527+0000","last_change":"2024-03-24T11:40:24.161914+0000","last_active":"2024-03-24T11:41:05.384527+0000","last_peered":"2024-03-24T11:41:05.384527+0000","last_clean":"2024-03-24T11:41:05.384527+0000","last_became_active":"2024-03-24T11:39:28.575371+0000","last_became_peered":"2024-03-24T11:39:28.575371+0000","last_unstale":"2024-03-24T11:41:05.384527+0000","last_undegraded":"2024-03-24T11:41:05.384527+0000","last_fullsized":"2024-03-24T11:41:05.384527+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:40:24.161610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:24.161610+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:48.775035+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":846,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327551+0000","last_change":"2024-03-24T11:40:35.292266+0000","last_active":"2024-03-24T11:41:03.327551+0000","last_peered":"2024-03-24T11:41:03.327551+0000","last_clean":"2024-03-24T11:41:03.327551+0000","last_became_active":"2024-03-24T11:40:35.291627+0000","last_became_peered":"2024-03-24T11:40:35.291627+0000","last_unstale":"2024-03-24T11:41:03.327551+0000","last_undegraded":"2024-03-24T11:41:03.327551+0000","last_fullsized":"2024-03-24T11:41:03.327551+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:47.913497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:47.913497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":191,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260079+0000","last_change":"2024-03-24T11:39:28.552174+0000","last_active":"2024-03-24T11:41:03.260079+0000","last_peered":"2024-03-24T11:41:03.260079+0000","last_clean":"2024-03-24T11:41:03.260079+0000","last_became_active":"2024-03-24T11:39:28.551550+0000","last_became_peered":"2024-03-24T11:39:28.551550+0000","last_unstale":"2024-03-24T11:41:03.260079+0000","last_undegraded":"2024-03-24T11:41:03.260079+0000","last_fullsized":"2024-03-24T11:41:03.260079+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_clean_scrub_stamp":"2024-03-24T11:39:06.116941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":855,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386349+0000","last_change":"2024-03-24T11:36:48.160280+0000","last_active":"2024-03-24T11:41:05.386349+0000","last_peered":"2024-03-24T11:41:05.386349+0000","last_clean":"2024-03-24T11:41:05.386349+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:41:05.386349+0000","last_undegraded":"2024-03-24T11:41:05.386349+0000","last_fullsized":"2024-03-24T11:41:05.386349+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:48.160164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:48.160164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"421'14","reported_seq":883,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.383870+0000","last_change":"2024-03-24T11:38:54.224427+0000","last_active":"2024-03-24T11:41:05.383870+0000","last_peered":"2024-03-24T11:41:05.383870+0000","last_clean":"2024-03-24T11:41:05.383870+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:41:05.383870+0000","last_undegraded":"2024-03-24T11:41:05.383870+0000","last_fullsized":"2024-03-24T11:41:05.383870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'10","last_scrub_stamp":"2024-03-24T11:38:54.222100+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:38:54.222100+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":890,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655912+0000","last_change":"2024-03-24T11:38:53.264039+0000","last_active":"2024-03-24T11:41:02.655912+0000","last_peered":"2024-03-24T11:41:02.655912+0000","last_clean":"2024-03-24T11:41:02.655912+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:41:02.655912+0000","last_undegraded":"2024-03-24T11:41:02.655912+0000","last_fullsized":"2024-03-24T11:41:02.655912+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:53.263761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.263761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":918,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260493+0000","last_change":"2024-03-24T11:41:01.285392+0000","last_active":"2024-03-24T11:41:03.260493+0000","last_peered":"2024-03-24T11:41:03.260493+0000","last_clean":"2024-03-24T11:41:03.260493+0000","last_became_active":"2024-03-24T11:39:28.572151+0000","last_became_peered":"2024-03-24T11:39:28.572151+0000","last_unstale":"2024-03-24T11:41:03.260493+0000","last_undegraded":"2024-03-24T11:41:03.260493+0000","last_fullsized":"2024-03-24T11:41:03.260493+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:01.285282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_clean_scrub_stamp":"2024-03-24T11:41:01.285282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:19.413936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":182,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.385645+0000","last_change":"2024-03-24T11:39:28.588074+0000","last_active":"2024-03-24T11:41:05.385645+0000","last_peered":"2024-03-24T11:41:05.385645+0000","last_clean":"2024-03-24T11:41:05.385645+0000","last_became_active":"2024-03-24T11:39:28.585224+0000","last_became_peered":"2024-03-24T11:39:28.585224+0000","last_unstale":"2024-03-24T11:41:05.385645+0000","last_undegraded":"2024-03-24T11:41:05.385645+0000","last_fullsized":"2024-03-24T11:41:05.385645+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:06.147813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:38:06.147813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"421'32","reported_seq":959,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261403+0000","last_change":"2024-03-24T11:40:56.233621+0000","last_active":"2024-03-24T11:41:03.261403+0000","last_peered":"2024-03-24T11:41:03.261403+0000","last_clean":"2024-03-24T11:41:03.261403+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:41:03.261403+0000","last_undegraded":"2024-03-24T11:41:03.261403+0000","last_fullsized":"2024-03-24T11:41:03.261403+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:40:56.233511+0000","last_deep_scrub":"87'16","last_deep_scrub_stamp":"2024-03-24T11:23:06.263941+0000","last_clean_scrub_stamp":"2024-03-24T11:40:56.233511+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:22.918300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":867,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.383780+0000","last_change":"2024-03-24T11:38:01.211155+0000","last_active":"2024-03-24T11:41:05.383780+0000","last_peered":"2024-03-24T11:41:05.383780+0000","last_clean":"2024-03-24T11:41:05.383780+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:41:05.383780+0000","last_undegraded":"2024-03-24T11:41:05.383780+0000","last_fullsized":"2024-03-24T11:41:05.383780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:01.211052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:38:01.211052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":890,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655852+0000","last_change":"2024-03-24T11:40:23.307812+0000","last_active":"2024-03-24T11:41:02.655852+0000","last_peered":"2024-03-24T11:41:02.655852+0000","last_clean":"2024-03-24T11:41:02.655852+0000","last_became_active":"2024-03-24T11:39:28.576207+0000","last_became_peered":"2024-03-24T11:39:28.576207+0000","last_unstale":"2024-03-24T11:41:02.655852+0000","last_undegraded":"2024-03-24T11:41:02.655852+0000","last_fullsized":"2024-03-24T11:41:02.655852+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:23.307700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:40:23.307700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:44.835484+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":891,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327386+0000","last_change":"2024-03-24T11:38:12.027553+0000","last_active":"2024-03-24T11:41:03.327386+0000","last_peered":"2024-03-24T11:41:03.327386+0000","last_clean":"2024-03-24T11:41:03.327386+0000","last_became_active":"2024-03-24T11:36:58.592355+0000","last_became_peered":"2024-03-24T11:36:58.592355+0000","last_unstale":"2024-03-24T11:41:03.327386+0000","last_undegraded":"2024-03-24T11:41:03.327386+0000","last_fullsized":"2024-03-24T11:41:03.327386+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.027430+0000","last_clean_scrub_stamp":"2024-03-24T11:38:12.027430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'30","reported_seq":977,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655697+0000","last_change":"2024-03-24T11:40:15.366849+0000","last_active":"2024-03-24T11:41:02.655697+0000","last_peered":"2024-03-24T11:41:02.655697+0000","last_clean":"2024-03-24T11:41:02.655697+0000","last_became_active":"2024-03-24T11:39:28.576661+0000","last_became_peered":"2024-03-24T11:39:28.576661+0000","last_unstale":"2024-03-24T11:41:02.655697+0000","last_undegraded":"2024-03-24T11:41:02.655697+0000","last_fullsized":"2024-03-24T11:41:02.655697+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:40:15.366706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:40:15.366706+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:26.739599+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":909,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326587+0000","last_change":"2024-03-24T11:37:52.677447+0000","last_active":"2024-03-24T11:41:03.326587+0000","last_peered":"2024-03-24T11:41:03.326587+0000","last_clean":"2024-03-24T11:41:03.326587+0000","last_became_active":"2024-03-24T11:36:58.593112+0000","last_became_peered":"2024-03-24T11:36:58.593112+0000","last_unstale":"2024-03-24T11:41:03.326587+0000","last_undegraded":"2024-03-24T11:41:03.326587+0000","last_fullsized":"2024-03-24T11:41:03.326587+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:37:52.677294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:52.677294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"429'11","reported_seq":1208,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261276+0000","last_change":"2024-03-24T11:40:52.183243+0000","last_active":"2024-03-24T11:41:03.261276+0000","last_peered":"2024-03-24T11:41:03.261276+0000","last_clean":"2024-03-24T11:41:03.261276+0000","last_became_active":"2024-03-24T11:39:28.579771+0000","last_became_peered":"2024-03-24T11:39:28.579771+0000","last_unstale":"2024-03-24T11:41:03.261276+0000","last_undegraded":"2024-03-24T11:41:03.261276+0000","last_fullsized":"2024-03-24T11:41:03.261276+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'11","last_scrub_stamp":"2024-03-24T11:40:52.183120+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:40:52.183120+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:11.717493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":551,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852681+0000","last_change":"2024-03-24T11:39:28.582106+0000","last_active":"2024-03-24T11:41:02.852681+0000","last_peered":"2024-03-24T11:41:02.852681+0000","last_clean":"2024-03-24T11:41:02.852681+0000","last_became_active":"2024-03-24T11:39:28.562081+0000","last_became_peered":"2024-03-24T11:39:28.562081+0000","last_unstale":"2024-03-24T11:41:02.852681+0000","last_undegraded":"2024-03-24T11:41:02.852681+0000","last_fullsized":"2024-03-24T11:41:02.852681+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:39:19.096734+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:28:27.754941+0000","last_clean_scrub_stamp":"2024-03-24T11:39:19.096734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"421'24","reported_seq":913,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386231+0000","last_change":"2024-03-24T11:36:59.961408+0000","last_active":"2024-03-24T11:41:05.386231+0000","last_peered":"2024-03-24T11:41:05.386231+0000","last_clean":"2024-03-24T11:41:05.386231+0000","last_became_active":"2024-03-24T11:36:58.591714+0000","last_became_peered":"2024-03-24T11:36:58.591714+0000","last_unstale":"2024-03-24T11:41:05.386231+0000","last_undegraded":"2024-03-24T11:41:05.386231+0000","last_fullsized":"2024-03-24T11:41:05.386231+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'18","last_scrub_stamp":"2024-03-24T11:36:59.961306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:36:59.961306+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":175,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326869+0000","last_change":"2024-03-24T11:37:50.620288+0000","last_active":"2024-03-24T11:41:03.326869+0000","last_peered":"2024-03-24T11:41:03.326869+0000","last_clean":"2024-03-24T11:41:03.326869+0000","last_became_active":"2024-03-24T11:36:58.597448+0000","last_became_peered":"2024-03-24T11:36:58.597448+0000","last_unstale":"2024-03-24T11:41:03.326869+0000","last_undegraded":"2024-03-24T11:41:03.326869+0000","last_fullsized":"2024-03-24T11:41:03.326869+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:50.620171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:37:50.620171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":184,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386373+0000","last_change":"2024-03-24T11:39:28.581090+0000","last_active":"2024-03-24T11:41:05.386373+0000","last_peered":"2024-03-24T11:41:05.386373+0000","last_clean":"2024-03-24T11:41:05.386373+0000","last_became_active":"2024-03-24T11:39:28.577496+0000","last_became_peered":"2024-03-24T11:39:28.577496+0000","last_unstale":"2024-03-24T11:41:05.386373+0000","last_undegraded":"2024-03-24T11:41:05.386373+0000","last_fullsized":"2024-03-24T11:41:05.386373+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.066063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.066063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":859,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386369+0000","last_change":"2024-03-24T11:40:54.412222+0000","last_active":"2024-03-24T11:41:05.386369+0000","last_peered":"2024-03-24T11:41:05.386369+0000","last_clean":"2024-03-24T11:41:05.386369+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:41:05.386369+0000","last_undegraded":"2024-03-24T11:41:05.386369+0000","last_fullsized":"2024-03-24T11:41:05.386369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:54.412091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:54.412091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:11.778682+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"421'39","reported_seq":978,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.330332+0000","last_change":"2024-03-24T11:41:03.330370+0000","last_active":"2024-03-24T11:41:03.330332+0000","last_peered":"2024-03-24T11:41:03.330332+0000","last_clean":"2024-03-24T11:41:03.330332+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:41:03.330332+0000","last_undegraded":"2024-03-24T11:41:03.330332+0000","last_fullsized":"2024-03-24T11:41:03.330332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:41:03.330265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:03.330265+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:31.510712+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":901,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655108+0000","last_change":"2024-03-24T11:39:28.573154+0000","last_active":"2024-03-24T11:41:02.655108+0000","last_peered":"2024-03-24T11:41:02.655108+0000","last_clean":"2024-03-24T11:41:02.655108+0000","last_became_active":"2024-03-24T11:39:28.572672+0000","last_became_peered":"2024-03-24T11:39:28.572672+0000","last_unstale":"2024-03-24T11:41:02.655108+0000","last_undegraded":"2024-03-24T11:41:02.655108+0000","last_fullsized":"2024-03-24T11:41:02.655108+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:05.380874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:39:05.380874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":887,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.327288+0000","last_change":"2024-03-24T11:37:43.684412+0000","last_active":"2024-03-24T11:41:03.327288+0000","last_peered":"2024-03-24T11:41:03.327288+0000","last_clean":"2024-03-24T11:41:03.327288+0000","last_became_active":"2024-03-24T11:36:58.596592+0000","last_became_peered":"2024-03-24T11:36:58.596592+0000","last_unstale":"2024-03-24T11:41:03.327288+0000","last_undegraded":"2024-03-24T11:41:03.327288+0000","last_fullsized":"2024-03-24T11:41:03.327288+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:37:43.684286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:37:43.684286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":184,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.261204+0000","last_change":"2024-03-24T11:40:16.249656+0000","last_active":"2024-03-24T11:41:03.261204+0000","last_peered":"2024-03-24T11:41:03.261204+0000","last_clean":"2024-03-24T11:41:03.261204+0000","last_became_active":"2024-03-24T11:36:58.575356+0000","last_became_peered":"2024-03-24T11:36:58.575356+0000","last_unstale":"2024-03-24T11:41:03.261204+0000","last_undegraded":"2024-03-24T11:41:03.261204+0000","last_fullsized":"2024-03-24T11:41:03.261204+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:16.249562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:40:16.249562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:34.813107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"421'9","reported_seq":542,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.385809+0000","last_change":"2024-03-24T11:41:04.222821+0000","last_active":"2024-03-24T11:41:05.385809+0000","last_peered":"2024-03-24T11:41:05.385809+0000","last_clean":"2024-03-24T11:41:05.385809+0000","last_became_active":"2024-03-24T11:36:58.586221+0000","last_became_peered":"2024-03-24T11:36:58.586221+0000","last_unstale":"2024-03-24T11:41:05.385809+0000","last_undegraded":"2024-03-24T11:41:05.385809+0000","last_fullsized":"2024-03-24T11:41:05.385809+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9","last_scrub_stamp":"2024-03-24T11:41:04.222695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:04.222695+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:16.064358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":895,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852906+0000","last_change":"2024-03-24T11:40:24.196213+0000","last_active":"2024-03-24T11:41:02.852906+0000","last_peered":"2024-03-24T11:41:02.852906+0000","last_clean":"2024-03-24T11:41:02.852906+0000","last_became_active":"2024-03-24T11:36:58.585301+0000","last_became_peered":"2024-03-24T11:36:58.585301+0000","last_unstale":"2024-03-24T11:41:02.852906+0000","last_undegraded":"2024-03-24T11:41:02.852906+0000","last_fullsized":"2024-03-24T11:41:02.852906+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:24.196131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:40:24.196131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:37.249324+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":191,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326842+0000","last_change":"2024-03-24T11:40:11.727255+0000","last_active":"2024-03-24T11:41:03.326842+0000","last_peered":"2024-03-24T11:41:03.326842+0000","last_clean":"2024-03-24T11:41:03.326842+0000","last_became_active":"2024-03-24T11:39:28.571768+0000","last_became_peered":"2024-03-24T11:39:28.571768+0000","last_unstale":"2024-03-24T11:41:03.326842+0000","last_undegraded":"2024-03-24T11:41:03.326842+0000","last_fullsized":"2024-03-24T11:41:03.326842+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:11.727141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.727141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:29.376135+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":878,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852524+0000","last_change":"2024-03-24T11:40:54.116829+0000","last_active":"2024-03-24T11:41:02.852524+0000","last_peered":"2024-03-24T11:41:02.852524+0000","last_clean":"2024-03-24T11:41:02.852524+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:41:02.852524+0000","last_undegraded":"2024-03-24T11:41:02.852524+0000","last_fullsized":"2024-03-24T11:41:02.852524+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:54.116658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:40:54.116658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:15.911672+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"421'21","reported_seq":556,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.385932+0000","last_change":"2024-03-24T11:38:53.240408+0000","last_active":"2024-03-24T11:41:05.385932+0000","last_peered":"2024-03-24T11:41:05.385932+0000","last_clean":"2024-03-24T11:41:05.385932+0000","last_became_active":"2024-03-24T11:36:58.580893+0000","last_became_peered":"2024-03-24T11:36:58.580893+0000","last_unstale":"2024-03-24T11:41:05.385932+0000","last_undegraded":"2024-03-24T11:41:05.385932+0000","last_fullsized":"2024-03-24T11:41:05.385932+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'15","last_scrub_stamp":"2024-03-24T11:38:53.238808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:38:53.238808+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":895,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.853017+0000","last_change":"2024-03-24T11:39:07.004625+0000","last_active":"2024-03-24T11:41:02.853017+0000","last_peered":"2024-03-24T11:41:02.853017+0000","last_clean":"2024-03-24T11:41:02.853017+0000","last_became_active":"2024-03-24T11:36:58.590154+0000","last_became_peered":"2024-03-24T11:36:58.590154+0000","last_unstale":"2024-03-24T11:41:02.853017+0000","last_undegraded":"2024-03-24T11:41:02.853017+0000","last_fullsized":"2024-03-24T11:41:02.853017+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:07.004520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:39:07.004520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":901,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852484+0000","last_change":"2024-03-24T11:40:55.105221+0000","last_active":"2024-03-24T11:41:02.852484+0000","last_peered":"2024-03-24T11:41:02.852484+0000","last_clean":"2024-03-24T11:41:02.852484+0000","last_became_active":"2024-03-24T11:39:28.562146+0000","last_became_peered":"2024-03-24T11:39:28.562146+0000","last_unstale":"2024-03-24T11:41:02.852484+0000","last_undegraded":"2024-03-24T11:41:02.852484+0000","last_fullsized":"2024-03-24T11:41:02.852484+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:55.105042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:40:55.105042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:59.762454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":884,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.654284+0000","last_change":"2024-03-24T11:40:17.412824+0000","last_active":"2024-03-24T11:41:02.654284+0000","last_peered":"2024-03-24T11:41:02.654284+0000","last_clean":"2024-03-24T11:41:02.654284+0000","last_became_active":"2024-03-24T11:39:28.548958+0000","last_became_peered":"2024-03-24T11:39:28.548958+0000","last_unstale":"2024-03-24T11:41:02.654284+0000","last_undegraded":"2024-03-24T11:41:02.654284+0000","last_fullsized":"2024-03-24T11:41:02.654284+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:17.412643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:17.412643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:31.388086+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'39","reported_seq":916,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.385695+0000","last_change":"2024-03-24T11:37:02.002116+0000","last_active":"2024-03-24T11:41:05.385695+0000","last_peered":"2024-03-24T11:41:05.385695+0000","last_clean":"2024-03-24T11:41:05.385695+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:41:05.385695+0000","last_undegraded":"2024-03-24T11:41:05.385695+0000","last_fullsized":"2024-03-24T11:41:05.385695+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'27","last_scrub_stamp":"2024-03-24T11:37:02.002019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:37:02.002019+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":869,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386461+0000","last_change":"2024-03-24T11:41:05.207842+0000","last_active":"2024-03-24T11:41:05.386461+0000","last_peered":"2024-03-24T11:41:05.386461+0000","last_clean":"2024-03-24T11:41:05.386461+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:41:05.386461+0000","last_undegraded":"2024-03-24T11:41:05.386461+0000","last_fullsized":"2024-03-24T11:41:05.386461+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:05.207762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:05.207762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:20.321055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":184,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260182+0000","last_change":"2024-03-24T11:39:11.310606+0000","last_active":"2024-03-24T11:41:03.260182+0000","last_peered":"2024-03-24T11:41:03.260182+0000","last_clean":"2024-03-24T11:41:03.260182+0000","last_became_active":"2024-03-24T11:36:58.584540+0000","last_became_peered":"2024-03-24T11:36:58.584540+0000","last_unstale":"2024-03-24T11:41:03.260182+0000","last_undegraded":"2024-03-24T11:41:03.260182+0000","last_fullsized":"2024-03-24T11:41:03.260182+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:39:11.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:39:11.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":852,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-24T11:41:05.386105+0000","last_change":"2024-03-24T11:39:28.584057+0000","last_active":"2024-03-24T11:41:05.386105+0000","last_peered":"2024-03-24T11:41:05.386105+0000","last_clean":"2024-03-24T11:41:05.386105+0000","last_became_active":"2024-03-24T11:39:28.580757+0000","last_became_peered":"2024-03-24T11:39:28.580757+0000","last_unstale":"2024-03-24T11:41:05.386105+0000","last_undegraded":"2024-03-24T11:41:05.386105+0000","last_fullsized":"2024-03-24T11:41:05.386105+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:36:53.164514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:36:53.164514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":586,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.260501+0000","last_change":"2024-03-24T11:39:17.302385+0000","last_active":"2024-03-24T11:41:03.260501+0000","last_peered":"2024-03-24T11:41:03.260501+0000","last_clean":"2024-03-24T11:41:03.260501+0000","last_became_active":"2024-03-24T11:36:58.580796+0000","last_became_peered":"2024-03-24T11:36:58.580796+0000","last_unstale":"2024-03-24T11:41:03.260501+0000","last_undegraded":"2024-03-24T11:41:03.260501+0000","last_fullsized":"2024-03-24T11:41:03.260501+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:39:17.302257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:39:17.302257+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":891,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.655760+0000","last_change":"2024-03-24T11:40:25.277893+0000","last_active":"2024-03-24T11:41:02.655760+0000","last_peered":"2024-03-24T11:41:02.655760+0000","last_clean":"2024-03-24T11:41:02.655760+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:41:02.655760+0000","last_undegraded":"2024-03-24T11:41:02.655760+0000","last_fullsized":"2024-03-24T11:41:02.655760+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:25.277603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:25.277603+0000","last_clean_scrub_stamp":"2024-03-24T11:40:25.277603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:50.379971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":896,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:03.326722+0000","last_change":"2024-03-24T11:40:57.792140+0000","last_active":"2024-03-24T11:41:03.326722+0000","last_peered":"2024-03-24T11:41:03.326722+0000","last_clean":"2024-03-24T11:41:03.326722+0000","last_became_active":"2024-03-24T11:39:28.557876+0000","last_became_peered":"2024-03-24T11:39:28.557876+0000","last_unstale":"2024-03-24T11:41:03.326722+0000","last_undegraded":"2024-03-24T11:41:03.326722+0000","last_fullsized":"2024-03-24T11:41:03.326722+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:57.792057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:40:57.792057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:42:27.070962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":189,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-24T11:41:02.852346+0000","last_change":"2024-03-24T11:40:11.312058+0000","last_active":"2024-03-24T11:41:02.852346+0000","last_peered":"2024-03-24T11:41:02.852346+0000","last_clean":"2024-03-24T11:41:02.852346+0000","last_became_active":"2024-03-24T11:39:28.546592+0000","last_became_peered":"2024-03-24T11:39:28.546592+0000","last_unstale":"2024-03-24T11:41:02.852346+0000","last_undegraded":"2024-03-24T11:41:02.852346+0000","last_fullsized":"2024-03-24T11:41:02.852346+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:40:11.311949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:40:11.311949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:41:20.458238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":628,"num_read_kb":623,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":5568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":3856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":13944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1031,"ondisk_log_size":1031,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365075,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":730696,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":93014456,"statfs":{"total":95995035648,"available":95246802944,"internally_reserved":0,"allocated":1241088,"data_stored":852338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":746977734},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575149,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934496,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92810656,"statfs":{"total":95995035648,"available":95038111744,"internally_reserved":0,"allocated":942080,"data_stored":517455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":955972037},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738674,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840072,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92905080,"statfs":{"total":95995035648,"available":95134801920,"internally_reserved":0,"allocated":913408,"data_stored":523750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411624,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722308,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93022844,"statfs":{"total":95995035648,"available":95255392256,"internally_reserved":0,"allocated":1171456,"data_stored":786265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149985,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880924,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864228,"statfs":{"total":95995035648,"available":95092969472,"internally_reserved":0,"allocated":1527808,"data_stored":1116109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":900528583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659416,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848992,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896160,"statfs":{"total":95995035648,"available":95125667840,"internally_reserved":0,"allocated":1527808,"data_stored":1114817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:41:08.124 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T11:41:08.124 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T11:41:08.125 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-24T11:41:08.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd in 7 2024-03-24T11:41:08.377 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:08.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:08 smithi181 ceph-mon[33998]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:41:08.448 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:08 smithi179 ceph-mon[29608]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:41:08.448 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:08 smithi179 ceph-mon[37907]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:41:09.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:09 smithi181 ceph-mon[33998]: pgmap v1034: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:09.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:09 smithi179 ceph-mon[29608]: pgmap v1034: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:09.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:09 smithi179 ceph-mon[37907]: pgmap v1034: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:10.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:10 smithi181 ceph-mon[33998]: 3.0 deep-scrub starts 2024-03-24T11:41:10.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:10 smithi181 ceph-mon[33998]: 3.0 deep-scrub ok 2024-03-24T11:41:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:10 smithi179 ceph-mon[37907]: 3.0 deep-scrub starts 2024-03-24T11:41:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:10 smithi179 ceph-mon[37907]: 3.0 deep-scrub ok 2024-03-24T11:41:10.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:10 smithi179 ceph-mon[29608]: 3.0 deep-scrub starts 2024-03-24T11:41:10.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:10 smithi179 ceph-mon[29608]: 3.0 deep-scrub ok 2024-03-24T11:41:10.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:10 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29604]: 2024-03-24T11:41:10.301+0000 7f36b2553700 -1 mon.a@0(leader).osd e466 definitely_dead 0 2024-03-24T11:41:11.158 INFO:teuthology.orchestra.run.smithi179.stderr:marked in osd.7. 2024-03-24T11:41:11.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:41:11.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:41:11.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:41:11.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:41:11.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:41:11.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:41:11.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:41:11.403 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:41:11.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:41:11.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/612838771' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-24T11:41:11.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-24T11:41:11.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:41:11.404 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:11 smithi181 ceph-mon[33998]: pgmap v1035: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:11.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:41:11.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/612838771' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:41:11.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[29608]: pgmap v1035: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/612838771' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-24T11:41:11.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-24T11:41:11.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:41:11.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:11 smithi179 ceph-mon[37907]: pgmap v1035: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:11.891 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-24T11:41:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[29608]: 5.0 scrub starts 2024-03-24T11:41:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[29608]: 5.0 scrub ok 2024-03-24T11:41:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-24T11:41:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[29608]: osdmap e467: 8 total, 8 up, 7 in 2024-03-24T11:41:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[29608]: 3.5 deep-scrub starts 2024-03-24T11:41:12.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[29608]: 3.5 deep-scrub ok 2024-03-24T11:41:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[37907]: 5.0 scrub starts 2024-03-24T11:41:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[37907]: 5.0 scrub ok 2024-03-24T11:41:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-24T11:41:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[37907]: osdmap e467: 8 total, 8 up, 7 in 2024-03-24T11:41:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[37907]: 3.5 deep-scrub starts 2024-03-24T11:41:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:12 smithi179 ceph-mon[37907]: 3.5 deep-scrub ok 2024-03-24T11:41:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:12 smithi181 ceph-mon[33998]: 5.0 scrub starts 2024-03-24T11:41:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:12 smithi181 ceph-mon[33998]: 5.0 scrub ok 2024-03-24T11:41:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:12 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-24T11:41:12.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:12 smithi181 ceph-mon[33998]: osdmap e467: 8 total, 8 up, 7 in 2024-03-24T11:41:12.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:12 smithi181 ceph-mon[33998]: 3.5 deep-scrub starts 2024-03-24T11:41:12.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:12 smithi181 ceph-mon[33998]: 3.5 deep-scrub ok 2024-03-24T11:41:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[29608]: 3.1a deep-scrub starts 2024-03-24T11:41:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[29608]: 3.1a deep-scrub ok 2024-03-24T11:41:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[29608]: osdmap e468: 8 total, 8 up, 7 in 2024-03-24T11:41:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[29608]: 5.15 scrub starts 2024-03-24T11:41:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[29608]: 5.15 scrub ok 2024-03-24T11:41:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[29608]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 718 B/s rd, 0 op/s 2024-03-24T11:41:13.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[29608]: osdmap e469: 8 total, 8 up, 7 in 2024-03-24T11:41:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[37907]: 3.1a deep-scrub starts 2024-03-24T11:41:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[37907]: 3.1a deep-scrub ok 2024-03-24T11:41:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[37907]: osdmap e468: 8 total, 8 up, 7 in 2024-03-24T11:41:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[37907]: 5.15 scrub starts 2024-03-24T11:41:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[37907]: 5.15 scrub ok 2024-03-24T11:41:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[37907]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 718 B/s rd, 0 op/s 2024-03-24T11:41:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:13 smithi179 ceph-mon[37907]: osdmap e469: 8 total, 8 up, 7 in 2024-03-24T11:41:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:13 smithi181 ceph-mon[33998]: 3.1a deep-scrub starts 2024-03-24T11:41:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:13 smithi181 ceph-mon[33998]: 3.1a deep-scrub ok 2024-03-24T11:41:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:13 smithi181 ceph-mon[33998]: osdmap e468: 8 total, 8 up, 7 in 2024-03-24T11:41:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:13 smithi181 ceph-mon[33998]: 5.15 scrub starts 2024-03-24T11:41:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:13 smithi181 ceph-mon[33998]: 5.15 scrub ok 2024-03-24T11:41:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:13 smithi181 ceph-mon[33998]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 718 B/s rd, 0 op/s 2024-03-24T11:41:13.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:13 smithi181 ceph-mon[33998]: osdmap e469: 8 total, 8 up, 7 in 2024-03-24T11:41:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 6.1b scrub starts 2024-03-24T11:41:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 6.1b scrub ok 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 3.12 scrub starts 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: 3.12 scrub ok 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[29608]: osdmap e470: 8 total, 8 up, 7 in 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 6.1b scrub starts 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 6.1b scrub ok 2024-03-24T11:41:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 3.12 scrub starts 2024-03-24T11:41:14.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: 3.12 scrub ok 2024-03-24T11:41:14.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:14 smithi179 ceph-mon[37907]: osdmap e470: 8 total, 8 up, 7 in 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 6.1b scrub starts 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 6.1b scrub ok 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 3.12 scrub starts 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: 3.12 scrub ok 2024-03-24T11:41:14.526 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-mon[33998]: osdmap e470: 8 total, 8 up, 7 in 2024-03-24T11:41:14.527 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:41:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:41:15.380 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:41:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: 3.14 scrub starts 2024-03-24T11:41:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: 3.14 scrub ok 2024-03-24T11:41:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:41:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:41:15.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:41:15.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:41:15.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:15.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: pgmap v1041: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 482 B/s, 14 objects/s recovering 2024-03-24T11:41:15.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:15.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:15 smithi181 ceph-mon[33998]: osdmap e471: 8 total, 8 up, 7 in 2024-03-24T11:41:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: 3.14 scrub starts 2024-03-24T11:41:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: 3.14 scrub ok 2024-03-24T11:41:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:41:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: pgmap v1041: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 482 B/s, 14 objects/s recovering 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[29608]: osdmap e471: 8 total, 8 up, 7 in 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: 3.14 scrub starts 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: 3.14 scrub ok 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:41:15.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:41:15.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:41:15.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:15.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: pgmap v1041: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 482 B/s, 14 objects/s recovering 2024-03-24T11:41:15.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:41:15.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:15 smithi179 ceph-mon[37907]: osdmap e471: 8 total, 8 up, 7 in 2024-03-24T11:41:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:16 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:41:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:16 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:41:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:16 smithi181 ceph-mon[33998]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:16 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:16 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:16.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:16 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:16.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[29608]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[37907]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:41:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:41:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:16 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:16.892 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-24T11:41:16.893 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-24T11:41:16.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd in 4 2024-03-24T11:41:17.142 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:17.588 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: 1.9 deep-scrub starts 2024-03-24T11:41:17.588 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: 1.9 deep-scrub ok 2024-03-24T11:41:17.588 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: 3.c scrub starts 2024-03-24T11:41:17.588 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: 3.c scrub ok 2024-03-24T11:41:17.588 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:17.588 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:17.588 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: pgmap v1043: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 74 KiB/s, 19 objects/s recovering 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[29608]: osdmap e472: 8 total, 8 up, 7 in 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: 1.9 deep-scrub starts 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: 1.9 deep-scrub ok 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: 3.c scrub starts 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: 3.c scrub ok 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: pgmap v1043: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 74 KiB/s, 19 objects/s recovering 2024-03-24T11:41:17.589 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:17 smithi179 ceph-mon[37907]: osdmap e472: 8 total, 8 up, 7 in 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: 1.9 deep-scrub starts 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: 1.9 deep-scrub ok 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: 3.c scrub starts 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: 3.c scrub ok 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: pgmap v1043: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 74 KiB/s, 19 objects/s recovering 2024-03-24T11:41:17.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:17 smithi181 ceph-mon[33998]: osdmap e472: 8 total, 8 up, 7 in 2024-03-24T11:41:18.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[37907]: 3.8 scrub starts 2024-03-24T11:41:18.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[37907]: 3.8 scrub ok 2024-03-24T11:41:18.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:18.612 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:18.612 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[37907]: osdmap e473: 8 total, 8 up, 7 in 2024-03-24T11:41:18.612 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[29608]: 3.8 scrub starts 2024-03-24T11:41:18.612 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[29608]: 3.8 scrub ok 2024-03-24T11:41:18.612 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:18.612 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:18.612 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:18 smithi179 ceph-mon[29608]: osdmap e473: 8 total, 8 up, 7 in 2024-03-24T11:41:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:18 smithi181 ceph-mon[33998]: 3.8 scrub starts 2024-03-24T11:41:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:18 smithi181 ceph-mon[33998]: 3.8 scrub ok 2024-03-24T11:41:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:18 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:18 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:18.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:18 smithi181 ceph-mon[33998]: osdmap e473: 8 total, 8 up, 7 in 2024-03-24T11:41:19.395 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29604]: 2024-03-24T11:41:19.065+0000 7f36b2553700 -1 mon.a@0(leader).osd e473 definitely_dead 0 2024-03-24T11:41:19.408 INFO:teuthology.orchestra.run.smithi179.stderr:marked in osd.4. 2024-03-24T11:41:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:19 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:41:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:19 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:41:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:19 smithi181 ceph-mon[33998]: pgmap v1046: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 73 KiB/s, 5 objects/s recovering 2024-03-24T11:41:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2991099642' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-24T11:41:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:19 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-24T11:41:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:19 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:19.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:41:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:41:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[29608]: pgmap v1046: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 73 KiB/s, 5 objects/s recovering 2024-03-24T11:41:19.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2991099642' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[37907]: pgmap v1046: 142 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 73 KiB/s, 5 objects/s recovering 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2991099642' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:41:19.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:20.212 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-24T11:41:20.212 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-24T11:41:20.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-24T11:41:20.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[29608]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[29608]: 3.4 scrub starts 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[29608]: 3.4 scrub ok 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[37907]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[37907]: 3.4 scrub starts 2024-03-24T11:41:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:20 smithi179 ceph-mon[37907]: 3.4 scrub ok 2024-03-24T11:41:20.496 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:20 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:41:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:20 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:41:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:20 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-24T11:41:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:20 smithi181 ceph-mon[33998]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T11:41:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:20 smithi181 ceph-mon[33998]: 3.4 scrub starts 2024-03-24T11:41:20.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:20 smithi181 ceph-mon[33998]: 3.4 scrub ok 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: 1.8 scrub starts 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: 1.8 scrub ok 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T11:41:21.601 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[29608]: pgmap v1049: 142 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: 1.8 scrub starts 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: 1.8 scrub ok 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T11:41:21.602 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:21 smithi179 ceph-mon[37907]: pgmap v1049: 142 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-24T11:41:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:41:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:41:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:21.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: 1.8 scrub starts 2024-03-24T11:41:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: 1.8 scrub ok 2024-03-24T11:41:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T11:41:21.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:21 smithi181 ceph-mon[33998]: pgmap v1049: 142 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-24T11:41:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[29608]: 2.1 deep-scrub starts 2024-03-24T11:41:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[29608]: 2.1 deep-scrub ok 2024-03-24T11:41:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[29608]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-03-24T11:41:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[29608]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[29608]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T11:41:22.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[29608]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[29608]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[37907]: 2.1 deep-scrub starts 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[37907]: 2.1 deep-scrub ok 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[37907]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[37907]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[37907]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[37907]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:22 smithi179 ceph-mon[37907]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:22 smithi181 ceph-mon[33998]: 2.1 deep-scrub starts 2024-03-24T11:41:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:22 smithi181 ceph-mon[33998]: 2.1 deep-scrub ok 2024-03-24T11:41:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:22 smithi181 ceph-mon[33998]: Health check failed: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded (PG_DEGRADED) 2024-03-24T11:41:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:22 smithi181 ceph-mon[33998]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:22 smithi181 ceph-mon[33998]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T11:41:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:22 smithi181 ceph-mon[33998]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:22.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:22 smithi181 ceph-mon[33998]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 83'128 2024-03-24T11:41:23.422 INFO:teuthology.orchestra.run.smithi179.stderr:set pool 1 pgp_num to 11 2024-03-24T11:41:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:41:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:41:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[37907]: pgmap v1051: 142 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-24T11:41:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1979980187' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:41:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:41:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:41:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:41:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[29608]: pgmap v1051: 142 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-24T11:41:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1979980187' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:41:23.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:23 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:41:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:23 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:41:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:23 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:41:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:23 smithi181 ceph-mon[33998]: pgmap v1051: 142 pgs: 1 active+recovery_wait+degraded, 2 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 5/594 objects degraded (0.842%); 0 B/s, 0 objects/s recovering 2024-03-24T11:41:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1979980187' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:41:23.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:23 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T11:41:24.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-24T11:41:24.407 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:24.416 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:41:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:24 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:41:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:24 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:41:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:24 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T11:41:24.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:24 smithi181 ceph-mon[33998]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T11:41:24.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:41:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:41:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:41:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T11:41:24.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[29608]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T11:41:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:41:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:41:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T11:41:24.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:24 smithi179 ceph-mon[37907]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T11:41:25.421 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:41:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:41:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:41:25.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:25.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[29608]: pgmap v1053: 142 pgs: 1 active+recovery_wait+degraded, 5 active+remapped+backfill_toofull, 136 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 5/594 objects degraded (0.842%); 452 B/s, 15 objects/s recovering 2024-03-24T11:41:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:41:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:41:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:25 smithi179 ceph-mon[37907]: pgmap v1053: 142 pgs: 1 active+recovery_wait+degraded, 5 active+remapped+backfill_toofull, 136 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 5/594 objects degraded (0.842%); 452 B/s, 15 objects/s recovering 2024-03-24T11:41:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:25 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:41:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:25 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:41:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:25.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:25 smithi181 ceph-mon[33998]: pgmap v1053: 142 pgs: 1 active+recovery_wait+degraded, 5 active+remapped+backfill_toofull, 136 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 5/594 objects degraded (0.842%); 452 B/s, 15 objects/s recovering 2024-03-24T11:41:26.307 INFO:teuthology.orchestra.run.smithi179.stdout:mon.a: {} 2024-03-24T11:41:26.319 INFO:teuthology.orchestra.run.smithi179.stdout:mon.b: {} 2024-03-24T11:41:26.328 INFO:teuthology.orchestra.run.smithi179.stdout:mon.c: {} 2024-03-24T11:41:26.329 INFO:teuthology.orchestra.run.smithi179.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:41:26.329 INFO:teuthology.orchestra.run.smithi179.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:41:26.329 INFO:teuthology.orchestra.run.smithi179.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T11:41:26.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:26 smithi179 ceph-mon[29608]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:26.649 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:26 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1721555259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:26.649 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:26 smithi179 ceph-mon[37907]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:26.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:26 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1721555259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:26 smithi181 ceph-mon[33998]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:26.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:26 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1721555259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:41:26.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T11:41:27.247 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:27.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:27 smithi179 ceph-mon[29608]: 3.1b scrub starts 2024-03-24T11:41:27.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:27 smithi179 ceph-mon[29608]: 3.1b scrub ok 2024-03-24T11:41:27.672 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:27 smithi179 ceph-mon[29608]: pgmap v1054: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 414 B/s, 13 objects/s recovering 2024-03-24T11:41:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:27 smithi179 ceph-mon[37907]: 3.1b scrub starts 2024-03-24T11:41:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:27 smithi179 ceph-mon[37907]: 3.1b scrub ok 2024-03-24T11:41:27.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:27 smithi179 ceph-mon[37907]: pgmap v1054: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 414 B/s, 13 objects/s recovering 2024-03-24T11:41:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:27 smithi181 ceph-mon[33998]: 3.1b scrub starts 2024-03-24T11:41:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:27 smithi181 ceph-mon[33998]: 3.1b scrub ok 2024-03-24T11:41:27.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:27 smithi181 ceph-mon[33998]: pgmap v1054: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 414 B/s, 13 objects/s recovering 2024-03-24T11:41:28.684 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:41:28.684 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:41:28.684 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:41:28.684 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 4.19 scrub starts 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 4.19 scrub ok 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[29608]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 4.19 scrub starts 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 4.19 scrub ok 2024-03-24T11:41:28.685 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:41:28.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:41:28.686 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:28 smithi179 ceph-mon[37907]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 4.19 scrub starts 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 4.19 scrub ok 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:41:28.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:28 smithi181 ceph-mon[33998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/594 objects degraded (0.842%), 1 pg degraded) 2024-03-24T11:41:29.441 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T11:41:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[29608]: 6.e deep-scrub starts 2024-03-24T11:41:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[29608]: 6.e deep-scrub ok 2024-03-24T11:41:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[29608]: pgmap v1055: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 316 B/s, 10 objects/s recovering 2024-03-24T11:41:29.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/968294388' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:29.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[37907]: 6.e deep-scrub starts 2024-03-24T11:41:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[37907]: 6.e deep-scrub ok 2024-03-24T11:41:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[37907]: pgmap v1055: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 316 B/s, 10 objects/s recovering 2024-03-24T11:41:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/968294388' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:29 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:29 smithi181 ceph-mon[33998]: 6.e deep-scrub starts 2024-03-24T11:41:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:29 smithi181 ceph-mon[33998]: 6.e deep-scrub ok 2024-03-24T11:41:29.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:29 smithi181 ceph-mon[33998]: pgmap v1055: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 316 B/s, 10 objects/s recovering 2024-03-24T11:41:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:29 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/968294388' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:29.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:29 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:30.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-24T11:41:30.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:30 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:41:30.459 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:30 smithi179 ceph-mon[29608]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T11:41:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:30 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:41:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:30 smithi179 ceph-mon[37907]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T11:41:30.472 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:30 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:41:30.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:30 smithi181 ceph-mon[33998]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T11:41:31.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:31 smithi179 ceph-mon[29608]: pgmap v1057: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 316 B/s, 10 objects/s recovering 2024-03-24T11:41:31.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:31 smithi179 ceph-mon[29608]: 5.2 scrub starts 2024-03-24T11:41:31.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:31 smithi179 ceph-mon[29608]: 5.2 scrub ok 2024-03-24T11:41:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:31 smithi179 ceph-mon[37907]: pgmap v1057: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 316 B/s, 10 objects/s recovering 2024-03-24T11:41:31.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:31 smithi179 ceph-mon[37907]: 5.2 scrub starts 2024-03-24T11:41:31.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:31 smithi179 ceph-mon[37907]: 5.2 scrub ok 2024-03-24T11:41:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:31 smithi181 ceph-mon[33998]: pgmap v1057: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 316 B/s, 10 objects/s recovering 2024-03-24T11:41:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:31 smithi181 ceph-mon[33998]: 5.2 scrub starts 2024-03-24T11:41:31.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:31 smithi181 ceph-mon[33998]: 5.2 scrub ok 2024-03-24T11:41:32.454 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-24T11:41:32.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[29608]: 3.d deep-scrub starts 2024-03-24T11:41:32.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[29608]: 3.d deep-scrub ok 2024-03-24T11:41:32.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3449949030' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:32.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:32.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[37907]: 3.d deep-scrub starts 2024-03-24T11:41:32.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[37907]: 3.d deep-scrub ok 2024-03-24T11:41:32.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3449949030' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:32.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:32 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:32 smithi181 ceph-mon[33998]: 3.d deep-scrub starts 2024-03-24T11:41:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:32 smithi181 ceph-mon[33998]: 3.d deep-scrub ok 2024-03-24T11:41:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:32 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3449949030' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:32.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:32 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T11:41:33.035 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T11:41:33.284 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[29608]: 4.1d scrub starts 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[29608]: 4.1d scrub ok 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[29608]: 6.9 deep-scrub starts 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[29608]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[29608]: pgmap v1059: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[29608]: 6.9 deep-scrub ok 2024-03-24T11:41:33.671 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[37907]: 4.1d scrub starts 2024-03-24T11:41:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[37907]: 4.1d scrub ok 2024-03-24T11:41:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[37907]: 6.9 deep-scrub starts 2024-03-24T11:41:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:41:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[37907]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T11:41:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[37907]: pgmap v1059: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-24T11:41:33.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:33 smithi179 ceph-mon[37907]: 6.9 deep-scrub ok 2024-03-24T11:41:33.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:33 smithi181 ceph-mon[33998]: 4.1d scrub starts 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:33 smithi181 ceph-mon[33998]: 4.1d scrub ok 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:33 smithi181 ceph-mon[33998]: 6.9 deep-scrub starts 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:33 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:33 smithi181 ceph-mon[33998]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:33 smithi181 ceph-mon[33998]: pgmap v1059: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-24T11:41:33.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:33 smithi181 ceph-mon[33998]: 6.9 deep-scrub ok 2024-03-24T11:41:34.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:41:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:41:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:41:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:41:34.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:41:34.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 4.0 deep-scrub starts 2024-03-24T11:41:34.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 4.0 deep-scrub ok 2024-03-24T11:41:34.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:41:34.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:41:34.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:41:34.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:41:34.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:34 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:41:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:41:34.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 4.0 deep-scrub starts 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 4.0 deep-scrub ok 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:41:34.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 4.0 deep-scrub starts 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 4.0 deep-scrub ok 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:41:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:34 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:35.300 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:41:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:35.469 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T11:41:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:41:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:41:35.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:41:35.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:41:35.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:35.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:41:35.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:41:35.714 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: pgmap v1060: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1947882003' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:41:35.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: pgmap v1060: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:35.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:35 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1947882003' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: pgmap v1060: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:35.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:35 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1947882003' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:36.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-24T11:41:36.476 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:36.734 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:36 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1947882003' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:41:36.734 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:36 smithi179 ceph-mon[29608]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T11:41:36.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:36 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1947882003' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:41:36.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:36 smithi179 ceph-mon[37907]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T11:41:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:36 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1947882003' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:41:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:36 smithi181 ceph-mon[33998]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T11:41:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:41:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[29608]: pgmap v1062: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[29608]: 6.4 deep-scrub starts 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[29608]: 6.4 deep-scrub ok 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:41:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:41:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[37907]: pgmap v1062: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-24T11:41:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[37907]: 6.4 deep-scrub starts 2024-03-24T11:41:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:37 smithi179 ceph-mon[37907]: 6.4 deep-scrub ok 2024-03-24T11:41:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:37 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:41:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:37 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:41:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:37 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:41:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:37 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:41:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:37 smithi181 ceph-mon[33998]: pgmap v1062: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-24T11:41:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:37 smithi181 ceph-mon[33998]: 6.4 deep-scrub starts 2024-03-24T11:41:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:37 smithi181 ceph-mon[33998]: 6.4 deep-scrub ok 2024-03-24T11:41:38.720 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[29608]: 4.9 deep-scrub starts 2024-03-24T11:41:38.720 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[29608]: 4.9 deep-scrub ok 2024-03-24T11:41:38.720 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[29608]: 5.5 deep-scrub starts 2024-03-24T11:41:38.720 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[29608]: 5.5 deep-scrub ok 2024-03-24T11:41:38.720 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[37907]: 4.9 deep-scrub starts 2024-03-24T11:41:38.720 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[37907]: 4.9 deep-scrub ok 2024-03-24T11:41:38.721 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[37907]: 5.5 deep-scrub starts 2024-03-24T11:41:38.721 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:38 smithi179 ceph-mon[37907]: 5.5 deep-scrub ok 2024-03-24T11:41:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:38 smithi181 ceph-mon[33998]: 4.9 deep-scrub starts 2024-03-24T11:41:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:38 smithi181 ceph-mon[33998]: 4.9 deep-scrub ok 2024-03-24T11:41:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:38 smithi181 ceph-mon[33998]: 5.5 deep-scrub starts 2024-03-24T11:41:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:38 smithi181 ceph-mon[33998]: 5.5 deep-scrub ok 2024-03-24T11:41:39.491 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-24T11:41:39.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:39 smithi179 ceph-mon[29608]: pgmap v1063: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:39.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:39 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2971648508' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:39.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:39 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:39.809 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:39 smithi179 ceph-mon[37907]: pgmap v1063: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:39.809 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:39 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2971648508' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:39.809 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:39 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:39 smithi181 ceph-mon[33998]: pgmap v1063: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:39 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2971648508' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:39 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T11:41:40.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T11:41:40.639 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:40.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 5.a deep-scrub starts 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 5.a deep-scrub ok 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 6.1d deep-scrub starts 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 4.a deep-scrub starts 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[29608]: 4.a deep-scrub ok 2024-03-24T11:41:40.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 5.a deep-scrub starts 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 5.a deep-scrub ok 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 6.1d deep-scrub starts 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 4.a deep-scrub starts 2024-03-24T11:41:40.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:40 smithi179 ceph-mon[37907]: 4.a deep-scrub ok 2024-03-24T11:41:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:41:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:41:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 5.a deep-scrub starts 2024-03-24T11:41:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 5.a deep-scrub ok 2024-03-24T11:41:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:41:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:41:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T11:41:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T11:41:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 6.1d deep-scrub starts 2024-03-24T11:41:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 4.a deep-scrub starts 2024-03-24T11:41:40.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:40 smithi181 ceph-mon[33998]: 4.a deep-scrub ok 2024-03-24T11:41:41.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: 4.13 scrub starts 2024-03-24T11:41:41.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: 4.13 scrub ok 2024-03-24T11:41:41.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: 6.1d deep-scrub ok 2024-03-24T11:41:41.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:41:41.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:41:41.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: pgmap v1065: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:41.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:41:41.819 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:41:41.819 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: 4.13 scrub starts 2024-03-24T11:41:41.819 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: 4.13 scrub ok 2024-03-24T11:41:41.819 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: 6.1d deep-scrub ok 2024-03-24T11:41:41.819 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:41:41.819 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:41:41.819 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: pgmap v1065: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:41.820 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:41:41.820 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:41 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:41:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: 4.13 scrub starts 2024-03-24T11:41:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: 4.13 scrub ok 2024-03-24T11:41:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: 6.1d deep-scrub ok 2024-03-24T11:41:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:41:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:41:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: pgmap v1065: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:41:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:41:41.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:41 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:41:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:42 smithi181 ceph-mon[33998]: 3.2 deep-scrub starts 2024-03-24T11:41:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:42 smithi181 ceph-mon[33998]: 3.2 deep-scrub ok 2024-03-24T11:41:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:42 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:41:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:42 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:41:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[29608]: 3.2 deep-scrub starts 2024-03-24T11:41:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[29608]: 3.2 deep-scrub ok 2024-03-24T11:41:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:41:42.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:41:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[37907]: 3.2 deep-scrub starts 2024-03-24T11:41:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[37907]: 3.2 deep-scrub ok 2024-03-24T11:41:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:41:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:42 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:41:43.502 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T11:41:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: 4.6 scrub starts 2024-03-24T11:41:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: 4.6 scrub ok 2024-03-24T11:41:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: pgmap v1066: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/935759612' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: 4.2 scrub starts 2024-03-24T11:41:43.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: 4.2 scrub ok 2024-03-24T11:41:43.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: 1.7 scrub starts 2024-03-24T11:41:43.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:43 smithi181 ceph-mon[33998]: 1.7 scrub ok 2024-03-24T11:41:43.950 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: 4.6 scrub starts 2024-03-24T11:41:43.950 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: 4.6 scrub ok 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: pgmap v1066: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/935759612' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: 4.2 scrub starts 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: 4.2 scrub ok 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: 1.7 scrub starts 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[29608]: 1.7 scrub ok 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: 4.6 scrub starts 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: 4.6 scrub ok 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: pgmap v1066: 142 pgs: 5 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:41:43.951 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/935759612' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:43.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:43.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: 4.2 scrub starts 2024-03-24T11:41:43.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: 4.2 scrub ok 2024-03-24T11:41:43.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: 1.7 scrub starts 2024-03-24T11:41:43.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:43 smithi179 ceph-mon[37907]: 1.7 scrub ok 2024-03-24T11:41:44.369 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-24T11:41:44.627 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:44 smithi181 ceph-mon[33998]: 1.4 scrub starts 2024-03-24T11:41:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:44 smithi181 ceph-mon[33998]: 1.4 scrub ok 2024-03-24T11:41:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:44 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:41:44.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:44 smithi181 ceph-mon[33998]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T11:41:44.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:41:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:41:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[37907]: 1.4 scrub starts 2024-03-24T11:41:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[37907]: 1.4 scrub ok 2024-03-24T11:41:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:41:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[37907]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T11:41:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[29608]: 1.4 scrub starts 2024-03-24T11:41:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[29608]: 1.4 scrub ok 2024-03-24T11:41:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:41:44.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:44 smithi179 ceph-mon[29608]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T11:41:45.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:41:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[29608]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[29608]: pgmap v1069: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:41:45.793 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:45.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[37907]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T11:41:45.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:45 smithi179 ceph-mon[37907]: pgmap v1069: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:41:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:45 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:41:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:45 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:41:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:45 smithi181 ceph-mon[33998]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T11:41:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:45 smithi181 ceph-mon[33998]: pgmap v1069: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:41:46.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:41:46.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[29608]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[37907]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:41:46.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:46 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:41:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:46 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:41:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:46 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:41:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:46 smithi181 ceph-mon[33998]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:46 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:41:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:46 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:41:47.528 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-24T11:41:47.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: 4.1a scrub starts 2024-03-24T11:41:47.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:47.860 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: 4.1a scrub ok 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: pgmap v1071: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3050667632' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: 4.1a scrub starts 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:47.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: 4.1a scrub ok 2024-03-24T11:41:47.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:41:47.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:41:47.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: pgmap v1071: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:47.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T11:41:47.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3050667632' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:47.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:47 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: 4.1a scrub starts 2024-03-24T11:41:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: 4.1a scrub ok 2024-03-24T11:41:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:41:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:41:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: pgmap v1071: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T11:41:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3050667632' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:47.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:47 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T11:41:48.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T11:41:48.542 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[37907]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[37907]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T11:41:48.794 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:41:48.795 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:41:48.795 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:41:48.795 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[29608]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T11:41:48.795 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:41:48.795 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:41:48.795 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:48 smithi179 ceph-mon[29608]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T11:41:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:48 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:41:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:48 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:41:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:48 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T11:41:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:48 smithi181 ceph-mon[33998]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T11:41:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:48 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:41:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:48 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:41:48.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:48 smithi181 ceph-mon[33998]: osdmap e486: 8 total, 8 up, 8 in 2024-03-24T11:41:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:41:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:41:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:41:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:41:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[37907]: pgmap v1073: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-24T11:41:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:41:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:41:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:41:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:41:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[29608]: pgmap v1073: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-24T11:41:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:49 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:41:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:49 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:41:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:49 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:41:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:49 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:41:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:49 smithi181 ceph-mon[33998]: pgmap v1073: 142 pgs: 1 peering, 4 active+remapped+backfill_toofull, 137 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-24T11:41:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:41:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:50 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:41:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:50 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:41:50.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:50 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:41:50.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:50 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:41:50.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:50 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:41:50.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:50 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:41:51.546 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T11:41:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: 3.1f deep-scrub starts 2024-03-24T11:41:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: 3.1f deep-scrub ok 2024-03-24T11:41:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: pgmap v1075: 142 pgs: 4 activating, 4 active+remapped+backfill_toofull, 134 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-03-24T11:41:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1270093652' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:51.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: 3.1f deep-scrub starts 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: 3.1f deep-scrub ok 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: pgmap v1075: 142 pgs: 4 activating, 4 active+remapped+backfill_toofull, 134 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1270093652' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:51.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:51 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: 3.1f deep-scrub starts 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: 3.1f deep-scrub ok 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: pgmap v1075: 142 pgs: 4 activating, 4 active+remapped+backfill_toofull, 134 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1270093652' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:51 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:52.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-24T11:41:52.621 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[29608]: 3.18 scrub starts 2024-03-24T11:41:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[29608]: 3.18 scrub ok 2024-03-24T11:41:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:41:52.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[29608]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T11:41:52.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[29608]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T11:41:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[37907]: 3.18 scrub starts 2024-03-24T11:41:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[37907]: 3.18 scrub ok 2024-03-24T11:41:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:41:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[37907]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T11:41:52.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:52 smithi179 ceph-mon[37907]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T11:41:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:52 smithi181 ceph-mon[33998]: 3.18 scrub starts 2024-03-24T11:41:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:52 smithi181 ceph-mon[33998]: 3.18 scrub ok 2024-03-24T11:41:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:52 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:41:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:52 smithi181 ceph-mon[33998]: osdmap e487: 8 total, 8 up, 8 in 2024-03-24T11:41:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:52 smithi181 ceph-mon[33998]: osdmap e488: 8 total, 8 up, 8 in 2024-03-24T11:41:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[29608]: 4.10 scrub starts 2024-03-24T11:41:53.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[29608]: 4.10 scrub ok 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[29608]: pgmap v1077: 142 pgs: 4 activating, 4 active+remapped+backfill_toofull, 134 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[29608]: 1.3 scrub starts 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[29608]: 1.3 scrub ok 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[37907]: 4.10 scrub starts 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[37907]: 4.10 scrub ok 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[37907]: pgmap v1077: 142 pgs: 4 activating, 4 active+remapped+backfill_toofull, 134 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[37907]: 1.3 scrub starts 2024-03-24T11:41:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:53 smithi179 ceph-mon[37907]: 1.3 scrub ok 2024-03-24T11:41:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:53 smithi181 ceph-mon[33998]: 4.10 scrub starts 2024-03-24T11:41:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:53 smithi181 ceph-mon[33998]: 4.10 scrub ok 2024-03-24T11:41:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:53 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:41:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:53 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:41:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:53 smithi181 ceph-mon[33998]: pgmap v1077: 142 pgs: 4 activating, 4 active+remapped+backfill_toofull, 134 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:53 smithi181 ceph-mon[33998]: 1.3 scrub starts 2024-03-24T11:41:53.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:53 smithi181 ceph-mon[33998]: 1.3 scrub ok 2024-03-24T11:41:54.576 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-24T11:41:54.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:41:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: 6.1c deep-scrub starts 2024-03-24T11:41:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: 6.1c deep-scrub ok 2024-03-24T11:41:54.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: 3.7 scrub starts 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: 3.7 scrub ok 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3513602665' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:54.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:54 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: 6.1c deep-scrub starts 2024-03-24T11:41:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: 6.1c deep-scrub ok 2024-03-24T11:41:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:41:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: 3.7 scrub starts 2024-03-24T11:41:54.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: 3.7 scrub ok 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3513602665' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: 6.1c deep-scrub starts 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: 6.1c deep-scrub ok 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: 3.7 scrub starts 2024-03-24T11:41:54.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: 3.7 scrub ok 2024-03-24T11:41:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:41:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: osdmap e489: 8 total, 8 up, 8 in 2024-03-24T11:41:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3513602665' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:54.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:54 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T11:41:55.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-24T11:41:55.428 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:55.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:41:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:41:55.801 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:41:55.801 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:41:55.801 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:41:55.801 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: pgmap v1080: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: 6.8 scrub starts 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: 6.8 scrub ok 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[29608]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:41:55.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: pgmap v1080: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: 6.8 scrub starts 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: 6.8 scrub ok 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:55.803 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:55 smithi179 ceph-mon[37907]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-03-24T11:41:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:41:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:41:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:41:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:41:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:41:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: pgmap v1080: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:41:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T11:41:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: osdmap e490: 8 total, 8 up, 8 in 2024-03-24T11:41:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: 6.8 scrub starts 2024-03-24T11:41:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: 6.8 scrub ok 2024-03-24T11:41:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:41:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:55 smithi181 ceph-mon[33998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-03-24T11:41:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:56 smithi181 ceph-mon[33998]: 3.11 scrub starts 2024-03-24T11:41:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:56 smithi181 ceph-mon[33998]: 3.11 scrub ok 2024-03-24T11:41:56.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:56 smithi179 ceph-mon[29608]: 3.11 scrub starts 2024-03-24T11:41:56.906 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:56 smithi179 ceph-mon[29608]: 3.11 scrub ok 2024-03-24T11:41:56.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:56 smithi179 ceph-mon[37907]: 3.11 scrub starts 2024-03-24T11:41:56.906 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:56 smithi179 ceph-mon[37907]: 3.11 scrub ok 2024-03-24T11:41:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:57 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:41:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:57 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:41:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:57 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:41:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:57 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:41:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:57 smithi181 ceph-mon[33998]: pgmap v1082: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:41:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:57 smithi181 ceph-mon[33998]: 6.15 scrub starts 2024-03-24T11:41:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:57 smithi181 ceph-mon[33998]: 6.15 scrub ok 2024-03-24T11:41:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:41:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:41:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:41:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:41:57.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[29608]: pgmap v1082: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[29608]: 6.15 scrub starts 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[29608]: 6.15 scrub ok 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[37907]: pgmap v1082: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[37907]: 6.15 scrub starts 2024-03-24T11:41:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:57 smithi179 ceph-mon[37907]: 6.15 scrub ok 2024-03-24T11:41:58.596 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.4 to 1 (10000) 2024-03-24T11:41:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:58 smithi181 ceph-mon[33998]: 4.14 scrub starts 2024-03-24T11:41:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:58 smithi181 ceph-mon[33998]: 4.14 scrub ok 2024-03-24T11:41:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:58 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1522971945' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:41:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:58 smithi179 ceph-mon[29608]: 4.14 scrub starts 2024-03-24T11:41:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:58 smithi179 ceph-mon[29608]: 4.14 scrub ok 2024-03-24T11:41:58.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:58 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1522971945' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:41:58.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:58 smithi179 ceph-mon[37907]: 4.14 scrub starts 2024-03-24T11:41:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:58 smithi179 ceph-mon[37907]: 4.14 scrub ok 2024-03-24T11:41:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:58 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1522971945' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:41:59.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-24T11:41:59.623 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:41:59.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[29608]: 4.11 scrub starts 2024-03-24T11:41:59.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[29608]: 4.11 scrub ok 2024-03-24T11:41:59.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[29608]: pgmap v1083: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:59.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1522971945' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:41:59.855 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[29608]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T11:41:59.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[37907]: 4.11 scrub starts 2024-03-24T11:41:59.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[37907]: 4.11 scrub ok 2024-03-24T11:41:59.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[37907]: pgmap v1083: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:59.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1522971945' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:41:59.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:41:59 smithi179 ceph-mon[37907]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T11:41:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:59 smithi181 ceph-mon[33998]: 4.11 scrub starts 2024-03-24T11:41:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:59 smithi181 ceph-mon[33998]: 4.11 scrub ok 2024-03-24T11:41:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:59 smithi181 ceph-mon[33998]: pgmap v1083: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:41:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:59 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1522971945' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:41:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:41:59 smithi181 ceph-mon[33998]: osdmap e491: 8 total, 8 up, 8 in 2024-03-24T11:42:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[29608]: 3.b scrub starts 2024-03-24T11:42:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[29608]: 3.b scrub ok 2024-03-24T11:42:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[29608]: 6.1e scrub starts 2024-03-24T11:42:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[29608]: 3.1c scrub starts 2024-03-24T11:42:00.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[29608]: 3.1c scrub ok 2024-03-24T11:42:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[29608]: 6.1e scrub ok 2024-03-24T11:42:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[37907]: 3.b scrub starts 2024-03-24T11:42:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[37907]: 3.b scrub ok 2024-03-24T11:42:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[37907]: 6.1e scrub starts 2024-03-24T11:42:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[37907]: 3.1c scrub starts 2024-03-24T11:42:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[37907]: 3.1c scrub ok 2024-03-24T11:42:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:00 smithi179 ceph-mon[37907]: 6.1e scrub ok 2024-03-24T11:42:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:00 smithi181 ceph-mon[33998]: 3.b scrub starts 2024-03-24T11:42:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:00 smithi181 ceph-mon[33998]: 3.b scrub ok 2024-03-24T11:42:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:00 smithi181 ceph-mon[33998]: 6.1e scrub starts 2024-03-24T11:42:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:00 smithi181 ceph-mon[33998]: 3.1c scrub starts 2024-03-24T11:42:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:00 smithi181 ceph-mon[33998]: 3.1c scrub ok 2024-03-24T11:42:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:00 smithi181 ceph-mon[33998]: 6.1e scrub ok 2024-03-24T11:42:01.602 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-24T11:42:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:01 smithi181 ceph-mon[33998]: pgmap v1085: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:42:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:01 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:42:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2116952261' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:42:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:01 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:42:01.929 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[29608]: pgmap v1085: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:42:01.929 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:42:01.929 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2116952261' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:42:01.929 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:42:01.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[37907]: pgmap v1085: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-24T11:42:01.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:42:01.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2116952261' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T11:42:01.930 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:01 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:42:02.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T11:42:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:02 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2116952261' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:42:02.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:02 smithi179 ceph-mon[29608]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T11:42:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:02 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2116952261' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:42:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:02 smithi179 ceph-mon[37907]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T11:42:02.750 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:02 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2116952261' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T11:42:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:02 smithi181 ceph-mon[33998]: osdmap e492: 8 total, 8 up, 8 in 2024-03-24T11:42:03.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[29608]: 4.e scrub starts 2024-03-24T11:42:03.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[29608]: 4.e scrub ok 2024-03-24T11:42:03.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[29608]: 5.16 scrub starts 2024-03-24T11:42:03.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[29608]: 5.16 scrub ok 2024-03-24T11:42:03.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[29608]: pgmap v1087: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-24T11:42:03.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[37907]: 4.e scrub starts 2024-03-24T11:42:03.872 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[37907]: 4.e scrub ok 2024-03-24T11:42:03.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[37907]: 5.16 scrub starts 2024-03-24T11:42:03.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[37907]: 5.16 scrub ok 2024-03-24T11:42:03.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:03 smithi179 ceph-mon[37907]: pgmap v1087: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-24T11:42:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:03 smithi181 ceph-mon[33998]: 4.e scrub starts 2024-03-24T11:42:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:03 smithi181 ceph-mon[33998]: 4.e scrub ok 2024-03-24T11:42:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:03 smithi181 ceph-mon[33998]: 5.16 scrub starts 2024-03-24T11:42:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:03 smithi181 ceph-mon[33998]: 5.16 scrub ok 2024-03-24T11:42:03.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:03 smithi181 ceph-mon[33998]: pgmap v1087: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-24T11:42:04.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:42:04.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: 5.17 scrub starts 2024-03-24T11:42:04.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: 5.17 scrub ok 2024-03-24T11:42:04.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: 3.19 scrub starts 2024-03-24T11:42:04.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: 3.19 scrub ok 2024-03-24T11:42:04.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:04.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: 5.11 scrub starts 2024-03-24T11:42:04.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: 5.11 scrub ok 2024-03-24T11:42:04.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:04 smithi181 ceph-mon[33998]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T11:42:04.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: 5.17 scrub starts 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: 5.17 scrub ok 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: 3.19 scrub starts 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: 3.19 scrub ok 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: 5.11 scrub starts 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: 5.11 scrub ok 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[29608]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: 5.17 scrub starts 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: 5.17 scrub ok 2024-03-24T11:42:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: 3.19 scrub starts 2024-03-24T11:42:04.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: 3.19 scrub ok 2024-03-24T11:42:04.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:04.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: 5.11 scrub starts 2024-03-24T11:42:04.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: 5.11 scrub ok 2024-03-24T11:42:04.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:04 smithi179 ceph-mon[37907]: osdmap e493: 8 total, 8 up, 8 in 2024-03-24T11:42:05.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:42:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:05.624 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T11:42:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:05 smithi181 ceph-mon[33998]: 5.1b scrub starts 2024-03-24T11:42:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:05 smithi181 ceph-mon[33998]: 5.1b scrub ok 2024-03-24T11:42:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:05 smithi181 ceph-mon[33998]: pgmap v1089: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:42:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:05 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4085734764' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:42:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[29608]: 5.1b scrub starts 2024-03-24T11:42:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[29608]: 5.1b scrub ok 2024-03-24T11:42:05.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[29608]: pgmap v1089: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:42:05.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4085734764' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:42:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[37907]: 5.1b scrub starts 2024-03-24T11:42:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[37907]: 5.1b scrub ok 2024-03-24T11:42:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[37907]: pgmap v1089: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:42:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:05 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4085734764' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:42:06.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-24T11:42:06.750 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:06.872 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4085734764' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:42:06.873 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:06 smithi179 ceph-mon[29608]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T11:42:06.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4085734764' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:42:06.873 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:06 smithi179 ceph-mon[37907]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T11:42:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4085734764' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:42:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:06 smithi181 ceph-mon[33998]: osdmap e494: 8 total, 8 up, 8 in 2024-03-24T11:42:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:07 smithi181 ceph-mon[33998]: pgmap v1091: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:07 smithi181 ceph-mon[33998]: 5.9 deep-scrub starts 2024-03-24T11:42:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:07 smithi181 ceph-mon[33998]: 5.9 deep-scrub ok 2024-03-24T11:42:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:07 smithi179 ceph-mon[29608]: pgmap v1091: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:07 smithi179 ceph-mon[29608]: 5.9 deep-scrub starts 2024-03-24T11:42:07.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:07 smithi179 ceph-mon[29608]: 5.9 deep-scrub ok 2024-03-24T11:42:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:07 smithi179 ceph-mon[37907]: pgmap v1091: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:07 smithi179 ceph-mon[37907]: 5.9 deep-scrub starts 2024-03-24T11:42:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:07 smithi179 ceph-mon[37907]: 5.9 deep-scrub ok 2024-03-24T11:42:09.393 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-24T11:42:09.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[29608]: pgmap v1092: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-24T11:42:09.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1699938552' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:42:09.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1699938552' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:42:09.683 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[29608]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T11:42:09.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[37907]: pgmap v1092: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-24T11:42:09.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1699938552' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:42:09.683 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1699938552' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:42:09.684 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:09 smithi179 ceph-mon[37907]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T11:42:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:09 smithi181 ceph-mon[33998]: pgmap v1092: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-24T11:42:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1699938552' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T11:42:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:09 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1699938552' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T11:42:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:09 smithi181 ceph-mon[33998]: osdmap e495: 8 total, 8 up, 8 in 2024-03-24T11:42:10.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T11:42:10.498 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 6.c scrub starts 2024-03-24T11:42:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 6.c scrub ok 2024-03-24T11:42:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 3.3 scrub starts 2024-03-24T11:42:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 3.3 scrub ok 2024-03-24T11:42:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 5.8 scrub starts 2024-03-24T11:42:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 5.8 scrub ok 2024-03-24T11:42:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 4.5 scrub starts 2024-03-24T11:42:10.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:10 smithi181 ceph-mon[33998]: 4.5 scrub ok 2024-03-24T11:42:10.934 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 6.c scrub starts 2024-03-24T11:42:10.934 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 6.c scrub ok 2024-03-24T11:42:10.934 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 3.3 scrub starts 2024-03-24T11:42:10.934 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 3.3 scrub ok 2024-03-24T11:42:10.934 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 5.8 scrub starts 2024-03-24T11:42:10.934 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 5.8 scrub ok 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 4.5 scrub starts 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[29608]: 4.5 scrub ok 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 6.c scrub starts 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 6.c scrub ok 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 3.3 scrub starts 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 3.3 scrub ok 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 5.8 scrub starts 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 5.8 scrub ok 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 4.5 scrub starts 2024-03-24T11:42:10.935 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:10 smithi179 ceph-mon[37907]: 4.5 scrub ok 2024-03-24T11:42:11.829 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[29608]: 6.b scrub starts 2024-03-24T11:42:11.829 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[29608]: 6.b scrub ok 2024-03-24T11:42:11.829 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[29608]: 1.1 scrub starts 2024-03-24T11:42:11.829 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[29608]: 1.1 scrub ok 2024-03-24T11:42:11.829 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[29608]: pgmap v1094: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:11.830 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[37907]: 6.b scrub starts 2024-03-24T11:42:11.830 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[37907]: 6.b scrub ok 2024-03-24T11:42:11.830 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[37907]: 1.1 scrub starts 2024-03-24T11:42:11.830 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[37907]: 1.1 scrub ok 2024-03-24T11:42:11.830 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:11 smithi179 ceph-mon[37907]: pgmap v1094: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:11 smithi181 ceph-mon[33998]: 6.b scrub starts 2024-03-24T11:42:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:11 smithi181 ceph-mon[33998]: 6.b scrub ok 2024-03-24T11:42:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:11 smithi181 ceph-mon[33998]: 1.1 scrub starts 2024-03-24T11:42:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:11 smithi181 ceph-mon[33998]: 1.1 scrub ok 2024-03-24T11:42:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:11 smithi181 ceph-mon[33998]: pgmap v1094: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:12.643 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T11:42:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:12 smithi181 ceph-mon[33998]: 6.2 scrub starts 2024-03-24T11:42:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:12 smithi181 ceph-mon[33998]: 6.2 scrub ok 2024-03-24T11:42:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:12 smithi181 ceph-mon[33998]: 6.1f scrub starts 2024-03-24T11:42:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:12 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1380937771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:12 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:12.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[37907]: 6.2 scrub starts 2024-03-24T11:42:12.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[37907]: 6.2 scrub ok 2024-03-24T11:42:12.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[37907]: 6.1f scrub starts 2024-03-24T11:42:12.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1380937771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:12.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[29608]: 6.2 scrub starts 2024-03-24T11:42:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[29608]: 6.2 scrub ok 2024-03-24T11:42:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[29608]: 6.1f scrub starts 2024-03-24T11:42:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1380937771' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:12.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:12 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:13.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-24T11:42:13.555 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:13.646 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[29608]: 6.1f scrub ok 2024-03-24T11:42:13.647 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[29608]: pgmap v1095: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:42:13.647 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:42:13.647 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[29608]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T11:42:13.647 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[29608]: 6.3 deep-scrub starts 2024-03-24T11:42:13.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[37907]: 6.1f scrub ok 2024-03-24T11:42:13.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[37907]: pgmap v1095: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:42:13.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:42:13.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[37907]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T11:42:13.900 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:13 smithi179 ceph-mon[37907]: 6.3 deep-scrub starts 2024-03-24T11:42:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:13 smithi181 ceph-mon[33998]: 6.1f scrub ok 2024-03-24T11:42:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:13 smithi181 ceph-mon[33998]: pgmap v1095: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:42:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:13 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:42:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:13 smithi181 ceph-mon[33998]: osdmap e496: 8 total, 8 up, 8 in 2024-03-24T11:42:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:13 smithi181 ceph-mon[33998]: 6.3 deep-scrub starts 2024-03-24T11:42:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:14 smithi181 ceph-mon[33998]: 4.d scrub starts 2024-03-24T11:42:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:14 smithi181 ceph-mon[33998]: 4.d scrub ok 2024-03-24T11:42:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:14 smithi181 ceph-mon[33998]: 6.3 deep-scrub ok 2024-03-24T11:42:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:14 smithi181 ceph-mon[33998]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T11:42:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:14 smithi181 ceph-mon[33998]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T11:42:14.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:42:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:42:14.938 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[29608]: 4.d scrub starts 2024-03-24T11:42:14.938 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[29608]: 4.d scrub ok 2024-03-24T11:42:14.938 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[29608]: 6.3 deep-scrub ok 2024-03-24T11:42:14.938 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[29608]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T11:42:14.939 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[29608]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T11:42:14.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[37907]: 4.d scrub starts 2024-03-24T11:42:14.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[37907]: 4.d scrub ok 2024-03-24T11:42:14.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[37907]: 6.3 deep-scrub ok 2024-03-24T11:42:14.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[37907]: osdmap e497: 8 total, 8 up, 8 in 2024-03-24T11:42:14.939 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:14 smithi179 ceph-mon[37907]: osdmap e498: 8 total, 8 up, 8 in 2024-03-24T11:42:15.193 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:42:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: 3.6 scrub starts 2024-03-24T11:42:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: 3.6 scrub ok 2024-03-24T11:42:15.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: pgmap v1099: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-03-24T11:42:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: 6.13 scrub starts 2024-03-24T11:42:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:42:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: 3.a scrub starts 2024-03-24T11:42:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: 3.a scrub ok 2024-03-24T11:42:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1989792800' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:15.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:15 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: 3.6 scrub starts 2024-03-24T11:42:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: 3.6 scrub ok 2024-03-24T11:42:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: pgmap v1099: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-03-24T11:42:15.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: 6.13 scrub starts 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: 3.a scrub starts 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: 3.a scrub ok 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1989792800' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: 3.6 scrub starts 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: 3.6 scrub ok 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: pgmap v1099: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 997 B/s rd, 0 op/s 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: 6.13 scrub starts 2024-03-24T11:42:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T11:42:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: 3.a scrub starts 2024-03-24T11:42:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: 3.a scrub ok 2024-03-24T11:42:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1989792800' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:15 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T11:42:16.403 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-24T11:42:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: 6.13 scrub ok 2024-03-24T11:42:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: 6.0 scrub starts 2024-03-24T11:42:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: 6.0 scrub ok 2024-03-24T11:42:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: 4.17 scrub starts 2024-03-24T11:42:16.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: 4.17 scrub ok 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: 6.13 scrub ok 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: 6.0 scrub starts 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: 6.0 scrub ok 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: 4.17 scrub starts 2024-03-24T11:42:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: 4.17 scrub ok 2024-03-24T11:42:16.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:16.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:42:16.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T11:42:16.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:16.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:16.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:16 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: 6.13 scrub ok 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: 6.0 scrub starts 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: 6.0 scrub ok 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: 4.17 scrub starts 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: 4.17 scrub ok 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: osdmap e499: 8 total, 8 up, 8 in 2024-03-24T11:42:16.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:42:16.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:42:16.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:16 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:42:17.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-24T11:42:17.259 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:17.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:17 smithi179 ceph-mon[29608]: pgmap v1101: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:17 smithi179 ceph-mon[37907]: pgmap v1101: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:18.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:17 smithi181 ceph-mon[33998]: pgmap v1101: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:18.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[29608]: 5.d scrub starts 2024-03-24T11:42:18.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[29608]: 5.d scrub ok 2024-03-24T11:42:18.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[29608]: 4.16 scrub starts 2024-03-24T11:42:18.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[29608]: 4.16 scrub ok 2024-03-24T11:42:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[37907]: 5.d scrub starts 2024-03-24T11:42:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[37907]: 5.d scrub ok 2024-03-24T11:42:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[37907]: 4.16 scrub starts 2024-03-24T11:42:18.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:18 smithi179 ceph-mon[37907]: 4.16 scrub ok 2024-03-24T11:42:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:18 smithi181 ceph-mon[33998]: 5.d scrub starts 2024-03-24T11:42:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:18 smithi181 ceph-mon[33998]: 5.d scrub ok 2024-03-24T11:42:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:18 smithi181 ceph-mon[33998]: 4.16 scrub starts 2024-03-24T11:42:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:18 smithi181 ceph-mon[33998]: 4.16 scrub ok 2024-03-24T11:42:19.402 INFO:teuthology.orchestra.run.smithi179.stderr:reweighted osd.7 to 1 (10000) 2024-03-24T11:42:19.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 3.9 scrub starts 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 3.9 scrub ok 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: pgmap v1102: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 3.8 scrub starts 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 3.8 scrub ok 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 3.f scrub starts 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 3.f scrub ok 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 6.19 scrub starts 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/4196312196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: 6.19 scrub ok 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:42:19.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[29608]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 3.9 scrub starts 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 3.9 scrub ok 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: pgmap v1102: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 3.8 scrub starts 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 3.8 scrub ok 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 3.f scrub starts 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 3.f scrub ok 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 6.19 scrub starts 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/4196312196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:19.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:19.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: 6.19 scrub ok 2024-03-24T11:42:19.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:42:19.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:19 smithi179 ceph-mon[37907]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T11:42:20.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 3.9 scrub starts 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 3.9 scrub ok 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: pgmap v1102: 142 pgs: 1 active+remapped+backfill_toofull, 1 peering, 140 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 3.8 scrub starts 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 3.8 scrub ok 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 3.f scrub starts 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 3.f scrub ok 2024-03-24T11:42:20.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 6.19 scrub starts 2024-03-24T11:42:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/4196312196' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: 6.19 scrub ok 2024-03-24T11:42:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:42:20.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:19 smithi181 ceph-mon[33998]: osdmap e500: 8 total, 8 up, 8 in 2024-03-24T11:42:20.354 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[29608]: 5.1f scrub starts 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[29608]: 5.1f scrub ok 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[29608]: 4.1b scrub starts 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[29608]: 4.1b scrub ok 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[37907]: 5.1f scrub starts 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[37907]: 5.1f scrub ok 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[37907]: 4.1b scrub starts 2024-03-24T11:42:20.750 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:20 smithi179 ceph-mon[37907]: 4.1b scrub ok 2024-03-24T11:42:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:20 smithi181 ceph-mon[33998]: 5.1f scrub starts 2024-03-24T11:42:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:20 smithi181 ceph-mon[33998]: 5.1f scrub ok 2024-03-24T11:42:21.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:20 smithi181 ceph-mon[33998]: 4.1b scrub starts 2024-03-24T11:42:21.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:20 smithi181 ceph-mon[33998]: 4.1b scrub ok 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: 5.19 scrub starts 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: 5.19 scrub ok 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: pgmap v1104: 142 pgs: 1 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: 4.12 scrub starts 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: 4.12 scrub ok 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: 6.5 scrub starts 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: 1.2 scrub starts 2024-03-24T11:42:21.778 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[29608]: 1.2 scrub ok 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: 5.19 scrub starts 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: 5.19 scrub ok 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: pgmap v1104: 142 pgs: 1 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: 4.12 scrub starts 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: 4.12 scrub ok 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: 6.5 scrub starts 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: 1.2 scrub starts 2024-03-24T11:42:21.779 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:21 smithi179 ceph-mon[37907]: 1.2 scrub ok 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: 5.19 scrub starts 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: 5.19 scrub ok 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: pgmap v1104: 142 pgs: 1 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: 4.12 scrub starts 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: 4.12 scrub ok 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: 6.5 scrub starts 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: 1.2 scrub starts 2024-03-24T11:42:22.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:21 smithi181 ceph-mon[33998]: 1.2 scrub ok 2024-03-24T11:42:22.683 INFO:teuthology.orchestra.run.smithi179.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-24T11:42:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[37907]: 6.5 scrub ok 2024-03-24T11:42:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[37907]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T11:42:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[37907]: 3.0 scrub starts 2024-03-24T11:42:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[37907]: 3.0 scrub ok 2024-03-24T11:42:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[37907]: 5.f scrub starts 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[37907]: 5.f scrub ok 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3673720994' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[29608]: 6.5 scrub ok 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[29608]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[29608]: 3.0 scrub starts 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[29608]: 3.0 scrub ok 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[29608]: 5.f scrub starts 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[29608]: 5.f scrub ok 2024-03-24T11:42:22.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:22 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3673720994' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:22 smithi181 ceph-mon[33998]: 6.5 scrub ok 2024-03-24T11:42:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:22 smithi181 ceph-mon[33998]: osdmap e501: 8 total, 8 up, 8 in 2024-03-24T11:42:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:22 smithi181 ceph-mon[33998]: 3.0 scrub starts 2024-03-24T11:42:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:22 smithi181 ceph-mon[33998]: 3.0 scrub ok 2024-03-24T11:42:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:22 smithi181 ceph-mon[33998]: 5.f scrub starts 2024-03-24T11:42:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:22 smithi181 ceph-mon[33998]: 5.f scrub ok 2024-03-24T11:42:23.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:22 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3673720994' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T11:42:23.280 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-24T11:42:23.281 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-24T11:42:23.281 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-24T11:42:23.281 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-24T11:42:23.281 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:42:23.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:42:23.528 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:23.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: pgmap v1106: 142 pgs: 1 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3673720994' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: 6.1b scrub starts 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: 1.5 scrub starts 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: 1.5 scrub ok 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: 4.1f deep-scrub starts 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[29608]: 4.1f deep-scrub ok 2024-03-24T11:42:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: pgmap v1106: 142 pgs: 1 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-03-24T11:42:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3673720994' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:42:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T11:42:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: 6.1b scrub starts 2024-03-24T11:42:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: 1.5 scrub starts 2024-03-24T11:42:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: 1.5 scrub ok 2024-03-24T11:42:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: 4.1f deep-scrub starts 2024-03-24T11:42:23.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:23 smithi179 ceph-mon[37907]: 4.1f deep-scrub ok 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: pgmap v1106: 142 pgs: 1 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3673720994' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: osdmap e502: 8 total, 8 up, 8 in 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: 6.1b scrub starts 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: 1.5 scrub starts 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: 1.5 scrub ok 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: 4.1f deep-scrub starts 2024-03-24T11:42:24.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:23 smithi181 ceph-mon[33998]: 4.1f deep-scrub ok 2024-03-24T11:42:24.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:42:24 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:42:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[29608]: 6.1b scrub ok 2024-03-24T11:42:24.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[29608]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[29608]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[37907]: 6.1b scrub ok 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[37907]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:42:24.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:24 smithi179 ceph-mon[37907]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T11:42:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:24 smithi181 ceph-mon[33998]: 6.1b scrub ok 2024-03-24T11:42:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:24 smithi181 ceph-mon[33998]: osdmap e503: 8 total, 8 up, 8 in 2024-03-24T11:42:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:24 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:42:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:24 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:42:25.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:24 smithi181 ceph-mon[33998]: osdmap e504: 8 total, 8 up, 8 in 2024-03-24T11:42:25.265 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:42:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:25.487 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:42:25.487 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":504,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:13:18.639983+0000","modified":"2024-03-24T11:42:24.389786+0000","last_up_change":"2024-03-24T11:18:40.079955+0000","last_in_change":"2024-03-24T11:41:19.388314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:16:40.018600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":359,"last_epoch_started":358,"last_epoch_clean":358,"source_version":"83'128","target_version":"83'128"},"last_change":"477","last_force_op_resend":"0","last_force_op_resend_prenautilus":"359","last_force_op_resend_preluminous":"357","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T11:18:51.329095+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T11:18:52.018113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:18:53.978924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:18:55.925755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:18:57.991833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14f300f0-7498-4ead-a87e-d9f38bdff21b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":501,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6803","nonce":1725503377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6805","nonce":1725503377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6809","nonce":1725503377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6807","nonce":1725503377}]},"public_addr":"172.21.15.179:6803/1725503377","cluster_addr":"172.21.15.179:6805/1725503377","heartbeat_back_addr":"172.21.15.179:6809/1725503377","heartbeat_front_addr":"172.21.15.179:6807/1725503377","state":["exists","up"]},{"osd":1,"uuid":"3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6811","nonce":2564847275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6813","nonce":2564847275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6817","nonce":2564847275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6815","nonce":2564847275}]},"public_addr":"172.21.15.179:6811/2564847275","cluster_addr":"172.21.15.179:6813/2564847275","heartbeat_back_addr":"172.21.15.179:6817/2564847275","heartbeat_front_addr":"172.21.15.179:6815/2564847275","state":["exists","up"]},{"osd":2,"uuid":"863b92b8-c68c-41a3-aed4-fd79e96d34dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6819","nonce":3297713842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6821","nonce":3297713842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6825","nonce":3297713842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6823","nonce":3297713842}]},"public_addr":"172.21.15.179:6819/3297713842","cluster_addr":"172.21.15.179:6821/3297713842","heartbeat_back_addr":"172.21.15.179:6825/3297713842","heartbeat_front_addr":"172.21.15.179:6823/3297713842","state":["exists","up"]},{"osd":3,"uuid":"c3d6b564-195d-4771-9aae-93a9e2eaba52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":474,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6827","nonce":944142471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6829","nonce":944142471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6833","nonce":944142471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6831","nonce":944142471}]},"public_addr":"172.21.15.179:6827/944142471","cluster_addr":"172.21.15.179:6829/944142471","heartbeat_back_addr":"172.21.15.179:6833/944142471","heartbeat_front_addr":"172.21.15.179:6831/944142471","state":["exists","up"]},{"osd":4,"uuid":"91bf5303-3a94-407d-aef1-e43bf0e16176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6801","nonce":3231660415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6803","nonce":3231660415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6807","nonce":3231660415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6805","nonce":3231660415}]},"public_addr":"172.21.15.181:6801/3231660415","cluster_addr":"172.21.15.181:6803/3231660415","heartbeat_back_addr":"172.21.15.181:6807/3231660415","heartbeat_front_addr":"172.21.15.181:6805/3231660415","state":["exists","up"]},{"osd":5,"uuid":"cba3f489-e8be-46f7-a281-e69a0ad08646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":474,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6809","nonce":775575782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6811","nonce":775575782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6815","nonce":775575782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6813","nonce":775575782}]},"public_addr":"172.21.15.181:6809/775575782","cluster_addr":"172.21.15.181:6811/775575782","heartbeat_back_addr":"172.21.15.181:6815/775575782","heartbeat_front_addr":"172.21.15.181:6813/775575782","state":["exists","up"]},{"osd":6,"uuid":"44a2c3a0-b704-4c10-a257-c54de47ee793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6817","nonce":2357509374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6819","nonce":2357509374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6823","nonce":2357509374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6821","nonce":2357509374}]},"public_addr":"172.21.15.181:6817/2357509374","cluster_addr":"172.21.15.181:6819/2357509374","heartbeat_back_addr":"172.21.15.181:6823/2357509374","heartbeat_front_addr":"172.21.15.181:6821/2357509374","state":["exists","up"]},{"osd":7,"uuid":"df60c4b4-d2f2-4e1f-8e1f-3169c403de00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":502,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6825","nonce":732039117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6827","nonce":732039117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6831","nonce":732039117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6829","nonce":732039117}]},"public_addr":"172.21.15.181:6825/732039117","cluster_addr":"172.21.15.181:6827/732039117","heartbeat_back_addr":"172.21.15.181:6831/732039117","heartbeat_front_addr":"172.21.15.181:6829/732039117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:17.175000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:04.948980+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:50.827689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:23.729376+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:54.378445+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:53.239679+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:19.885238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:46.161659+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.6","osds":[5,2,3]}],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":6}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":3,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.15","mappings":[{"from":3,"to":0}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":3,"to":2}]},{"pgid":"5.1b","mappings":[{"from":0,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1519226585":"2024-03-25T11:19:31.961858+0000","172.21.15.179:6800/2865046672":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/2472348284":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/2047439415":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6800/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/3185072343":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/3344305045":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2748363950":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/2066930919":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/2108875546":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6801/2865046672":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/4087802319":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/883221702":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/4255117010":"2024-03-25T11:19:31.961858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:42:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:25.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[29608]: pgmap v1110: 142 pgs: 20 peering, 122 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[29608]: 4.4 scrub starts 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[29608]: 4.4 scrub ok 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[29608]: Health check failed: Reduced data availability: 18 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[29608]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2963532208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[37907]: pgmap v1110: 142 pgs: 20 peering, 122 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[37907]: 4.4 scrub starts 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[37907]: 4.4 scrub ok 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[37907]: Health check failed: Reduced data availability: 18 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-24T11:42:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[37907]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:42:25.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:25 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2963532208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:42:26.081 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:42:26.082 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:25 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:25 smithi181 ceph-mon[33998]: pgmap v1110: 142 pgs: 20 peering, 122 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:25 smithi181 ceph-mon[33998]: 4.4 scrub starts 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:25 smithi181 ceph-mon[33998]: 4.4 scrub ok 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:25 smithi181 ceph-mon[33998]: Health check failed: Reduced data availability: 18 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:25 smithi181 ceph-mon[33998]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T11:42:26.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:25 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2963532208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:42:26.323 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:26.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:26 smithi179 ceph-mon[29608]: 6.14 scrub starts 2024-03-24T11:42:26.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:26 smithi179 ceph-mon[29608]: 6.14 scrub ok 2024-03-24T11:42:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:26 smithi179 ceph-mon[37907]: 6.14 scrub starts 2024-03-24T11:42:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:26 smithi179 ceph-mon[37907]: 6.14 scrub ok 2024-03-24T11:42:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:26 smithi181 ceph-mon[33998]: 6.14 scrub starts 2024-03-24T11:42:27.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:26 smithi181 ceph-mon[33998]: 6.14 scrub ok 2024-03-24T11:42:27.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[29608]: pgmap v1111: 142 pgs: 20 peering, 122 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:27.857 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[29608]: 5.12 scrub starts 2024-03-24T11:42:27.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[29608]: 5.12 scrub ok 2024-03-24T11:42:27.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[29608]: 5.15 deep-scrub starts 2024-03-24T11:42:27.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[29608]: 5.15 deep-scrub ok 2024-03-24T11:42:27.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[29608]: 5.6 scrub starts 2024-03-24T11:42:27.858 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[29608]: 5.6 scrub ok 2024-03-24T11:42:27.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[37907]: pgmap v1111: 142 pgs: 20 peering, 122 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:27.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[37907]: 5.12 scrub starts 2024-03-24T11:42:27.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[37907]: 5.12 scrub ok 2024-03-24T11:42:27.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[37907]: 5.15 deep-scrub starts 2024-03-24T11:42:27.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[37907]: 5.15 deep-scrub ok 2024-03-24T11:42:27.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[37907]: 5.6 scrub starts 2024-03-24T11:42:27.860 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:27 smithi179 ceph-mon[37907]: 5.6 scrub ok 2024-03-24T11:42:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:27 smithi181 ceph-mon[33998]: pgmap v1111: 142 pgs: 20 peering, 122 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:27 smithi181 ceph-mon[33998]: 5.12 scrub starts 2024-03-24T11:42:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:27 smithi181 ceph-mon[33998]: 5.12 scrub ok 2024-03-24T11:42:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:27 smithi181 ceph-mon[33998]: 5.15 deep-scrub starts 2024-03-24T11:42:28.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:27 smithi181 ceph-mon[33998]: 5.15 deep-scrub ok 2024-03-24T11:42:28.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:27 smithi181 ceph-mon[33998]: 5.6 scrub starts 2024-03-24T11:42:28.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:27 smithi181 ceph-mon[33998]: 5.6 scrub ok 2024-03-24T11:42:28.590 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:42:28.590 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":504,"fsid":"6430538c-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:13:18.639983+0000","modified":"2024-03-24T11:42:24.389786+0000","last_up_change":"2024-03-24T11:18:40.079955+0000","last_in_change":"2024-03-24T11:41:19.388314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:16:40.018600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":359,"last_epoch_started":358,"last_epoch_clean":358,"source_version":"83'128","target_version":"83'128"},"last_change":"477","last_force_op_resend":"0","last_force_op_resend_prenautilus":"359","last_force_op_resend_preluminous":"357","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T11:18:51.329095+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T11:18:52.018113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:18:53.978924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:18:55.925755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:18:57.991833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14f300f0-7498-4ead-a87e-d9f38bdff21b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":501,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6803","nonce":1725503377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6805","nonce":1725503377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6809","nonce":1725503377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":1725503377},{"type":"v1","addr":"172.21.15.179:6807","nonce":1725503377}]},"public_addr":"172.21.15.179:6803/1725503377","cluster_addr":"172.21.15.179:6805/1725503377","heartbeat_back_addr":"172.21.15.179:6809/1725503377","heartbeat_front_addr":"172.21.15.179:6807/1725503377","state":["exists","up"]},{"osd":1,"uuid":"3b25fe01-a3cc-4f9f-8e65-0bdd1d56a667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6811","nonce":2564847275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6813","nonce":2564847275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6817","nonce":2564847275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2564847275},{"type":"v1","addr":"172.21.15.179:6815","nonce":2564847275}]},"public_addr":"172.21.15.179:6811/2564847275","cluster_addr":"172.21.15.179:6813/2564847275","heartbeat_back_addr":"172.21.15.179:6817/2564847275","heartbeat_front_addr":"172.21.15.179:6815/2564847275","state":["exists","up"]},{"osd":2,"uuid":"863b92b8-c68c-41a3-aed4-fd79e96d34dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6819","nonce":3297713842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6821","nonce":3297713842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6825","nonce":3297713842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3297713842},{"type":"v1","addr":"172.21.15.179:6823","nonce":3297713842}]},"public_addr":"172.21.15.179:6819/3297713842","cluster_addr":"172.21.15.179:6821/3297713842","heartbeat_back_addr":"172.21.15.179:6825/3297713842","heartbeat_front_addr":"172.21.15.179:6823/3297713842","state":["exists","up"]},{"osd":3,"uuid":"c3d6b564-195d-4771-9aae-93a9e2eaba52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":474,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6827","nonce":944142471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6829","nonce":944142471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6833","nonce":944142471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":944142471},{"type":"v1","addr":"172.21.15.179:6831","nonce":944142471}]},"public_addr":"172.21.15.179:6827/944142471","cluster_addr":"172.21.15.179:6829/944142471","heartbeat_back_addr":"172.21.15.179:6833/944142471","heartbeat_front_addr":"172.21.15.179:6831/944142471","state":["exists","up"]},{"osd":4,"uuid":"91bf5303-3a94-407d-aef1-e43bf0e16176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6801","nonce":3231660415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6803","nonce":3231660415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6807","nonce":3231660415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3231660415},{"type":"v1","addr":"172.21.15.181:6805","nonce":3231660415}]},"public_addr":"172.21.15.181:6801/3231660415","cluster_addr":"172.21.15.181:6803/3231660415","heartbeat_back_addr":"172.21.15.181:6807/3231660415","heartbeat_front_addr":"172.21.15.181:6805/3231660415","state":["exists","up"]},{"osd":5,"uuid":"cba3f489-e8be-46f7-a281-e69a0ad08646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":474,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6809","nonce":775575782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6811","nonce":775575782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6815","nonce":775575782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":775575782},{"type":"v1","addr":"172.21.15.181:6813","nonce":775575782}]},"public_addr":"172.21.15.181:6809/775575782","cluster_addr":"172.21.15.181:6811/775575782","heartbeat_back_addr":"172.21.15.181:6815/775575782","heartbeat_front_addr":"172.21.15.181:6813/775575782","state":["exists","up"]},{"osd":6,"uuid":"44a2c3a0-b704-4c10-a257-c54de47ee793","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6817","nonce":2357509374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6819","nonce":2357509374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6823","nonce":2357509374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2357509374},{"type":"v1","addr":"172.21.15.181:6821","nonce":2357509374}]},"public_addr":"172.21.15.181:6817/2357509374","cluster_addr":"172.21.15.181:6819/2357509374","heartbeat_back_addr":"172.21.15.181:6823/2357509374","heartbeat_front_addr":"172.21.15.181:6821/2357509374","state":["exists","up"]},{"osd":7,"uuid":"df60c4b4-d2f2-4e1f-8e1f-3169c403de00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":502,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6825","nonce":732039117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6827","nonce":732039117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6831","nonce":732039117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":732039117},{"type":"v1","addr":"172.21.15.181:6829","nonce":732039117}]},"public_addr":"172.21.15.181:6825/732039117","cluster_addr":"172.21.15.181:6827/732039117","heartbeat_back_addr":"172.21.15.181:6831/732039117","heartbeat_front_addr":"172.21.15.181:6829/732039117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:17.175000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:04.948980+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:50.827689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:42:23.729376+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:54.378445+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:53.239679+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:19.885238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:41:46.161659+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.6","osds":[5,2,3]}],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":6}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":3,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.15","mappings":[{"from":3,"to":0}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":3,"to":2}]},{"pgid":"5.1b","mappings":[{"from":0,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1519226585":"2024-03-25T11:19:31.961858+0000","172.21.15.179:6800/2865046672":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/2472348284":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/2047439415":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6800/3169437":"2024-03-25T11:13:51.373673+0000","172.21.15.179:6801/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/3185072343":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/3344305045":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/2748363950":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6800/243537818":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/2066930919":"2024-03-25T11:14:17.688945+0000","172.21.15.179:0/2108875546":"2024-03-25T11:14:17.688945+0000","172.21.15.179:6801/2865046672":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/4087802319":"2024-03-25T11:19:31.961858+0000","172.21.15.179:0/883221702":"2024-03-25T11:13:51.373673+0000","172.21.15.179:0/4255117010":"2024-03-25T11:19:31.961858+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:42:28.902 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 3.e scrub starts 2024-03-24T11:42:28.902 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 3.e scrub ok 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 3.1a scrub starts 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 3.1a scrub ok 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 3.5 scrub starts 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 3.5 scrub ok 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 4.8 scrub starts 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 4.8 scrub ok 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 5.b scrub starts 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: 5.b scrub ok 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1119105873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 3.e scrub starts 2024-03-24T11:42:28.903 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 3.e scrub ok 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 3.1a scrub starts 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 3.1a scrub ok 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 3.5 scrub starts 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 3.5 scrub ok 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 4.8 scrub starts 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 4.8 scrub ok 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 5.b scrub starts 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: 5.b scrub ok 2024-03-24T11:42:28.904 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:28 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1119105873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 3.e scrub starts 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 3.e scrub ok 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 3.1a scrub starts 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 3.1a scrub ok 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 3.5 scrub starts 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 3.5 scrub ok 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 4.8 scrub starts 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 4.8 scrub ok 2024-03-24T11:42:29.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 5.b scrub starts 2024-03-24T11:42:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: 5.b scrub ok 2024-03-24T11:42:29.153 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:28 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1119105873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:42:29.295 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:42:29.295 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:42:29.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:42:29.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T11:42:29.296 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T11:42:29.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T11:42:29.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T11:42:29.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T11:42:29.858 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:29.859 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:29.860 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:29.864 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:29.865 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:29.865 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:29.866 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:29.873 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:30.000 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[29608]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:30.000 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[29608]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 18 pgs inactive, 19 pgs peering) 2024-03-24T11:42:30.000 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[29608]: Cluster is now healthy 2024-03-24T11:42:30.000 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[29608]: 5.3 scrub starts 2024-03-24T11:42:30.001 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[29608]: 5.3 scrub ok 2024-03-24T11:42:30.001 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[37907]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:30.001 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[37907]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 18 pgs inactive, 19 pgs peering) 2024-03-24T11:42:30.001 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[37907]: Cluster is now healthy 2024-03-24T11:42:30.001 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[37907]: 5.3 scrub starts 2024-03-24T11:42:30.001 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:29 smithi179 ceph-mon[37907]: 5.3 scrub ok 2024-03-24T11:42:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:29 smithi181 ceph-mon[33998]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:29 smithi181 ceph-mon[33998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 18 pgs inactive, 19 pgs peering) 2024-03-24T11:42:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:29 smithi181 ceph-mon[33998]: Cluster is now healthy 2024-03-24T11:42:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:29 smithi181 ceph-mon[33998]: 5.3 scrub starts 2024-03-24T11:42:30.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:29 smithi181 ceph-mon[33998]: 5.3 scrub ok 2024-03-24T11:42:31.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:30 smithi179 ceph-mon[29608]: 6.17 scrub starts 2024-03-24T11:42:31.052 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:30 smithi179 ceph-mon[29608]: 6.17 scrub ok 2024-03-24T11:42:31.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:30 smithi179 ceph-mon[37907]: 6.17 scrub starts 2024-03-24T11:42:31.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:30 smithi179 ceph-mon[37907]: 6.17 scrub ok 2024-03-24T11:42:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:30 smithi181 ceph-mon[33998]: 6.17 scrub starts 2024-03-24T11:42:31.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:30 smithi181 ceph-mon[33998]: 6.17 scrub ok 2024-03-24T11:42:32.046 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:31 smithi179 ceph-mon[29608]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:32.046 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:31 smithi179 ceph-mon[29608]: 6.7 scrub starts 2024-03-24T11:42:32.046 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:31 smithi179 ceph-mon[29608]: 6.7 scrub ok 2024-03-24T11:42:32.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:31 smithi179 ceph-mon[37907]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:32.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:31 smithi179 ceph-mon[37907]: 6.7 scrub starts 2024-03-24T11:42:32.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:31 smithi179 ceph-mon[37907]: 6.7 scrub ok 2024-03-24T11:42:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:31 smithi181 ceph-mon[33998]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail 2024-03-24T11:42:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:31 smithi181 ceph-mon[33998]: 6.7 scrub starts 2024-03-24T11:42:32.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:31 smithi181 ceph-mon[33998]: 6.7 scrub ok 2024-03-24T11:42:33.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:32 smithi181 ceph-mon[33998]: 1.0 scrub starts 2024-03-24T11:42:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:32 smithi181 ceph-mon[33998]: 1.0 scrub ok 2024-03-24T11:42:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:32 smithi181 ceph-mon[33998]: 1.a scrub starts 2024-03-24T11:42:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:32 smithi181 ceph-mon[33998]: 1.a scrub ok 2024-03-24T11:42:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:32 smithi181 ceph-mon[33998]: 2.1 scrub starts 2024-03-24T11:42:33.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:32 smithi181 ceph-mon[33998]: 2.1 scrub ok 2024-03-24T11:42:33.080 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[29608]: 1.0 scrub starts 2024-03-24T11:42:33.080 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[29608]: 1.0 scrub ok 2024-03-24T11:42:33.080 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[29608]: 1.a scrub starts 2024-03-24T11:42:33.080 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[29608]: 1.a scrub ok 2024-03-24T11:42:33.080 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[29608]: 2.1 scrub starts 2024-03-24T11:42:33.080 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[29608]: 2.1 scrub ok 2024-03-24T11:42:33.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[37907]: 1.0 scrub starts 2024-03-24T11:42:33.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[37907]: 1.0 scrub ok 2024-03-24T11:42:33.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[37907]: 1.a scrub starts 2024-03-24T11:42:33.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[37907]: 1.a scrub ok 2024-03-24T11:42:33.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[37907]: 2.1 scrub starts 2024-03-24T11:42:33.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:32 smithi179 ceph-mon[37907]: 2.1 scrub ok 2024-03-24T11:42:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:42:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:42:33.959 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[29608]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[29608]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[29608]: 6.a scrub starts 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[29608]: 6.a scrub ok 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[37907]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[37907]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[37907]: 6.a scrub starts 2024-03-24T11:42:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:33 smithi179 ceph-mon[37907]: 6.a scrub ok 2024-03-24T11:42:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:33 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:42:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:33 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:42:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:33 smithi181 ceph-mon[33998]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-03-24T11:42:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:33 smithi181 ceph-mon[33998]: osdmap e505: 8 total, 8 up, 8 in 2024-03-24T11:42:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:33 smithi181 ceph-mon[33998]: 6.a scrub starts 2024-03-24T11:42:34.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:33 smithi181 ceph-mon[33998]: 6.a scrub ok 2024-03-24T11:42:34.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:42:34 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:42:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: 4.1c scrub starts 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: 4.1c scrub ok 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: 3.1 scrub starts 2024-03-24T11:42:35.287 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[29608]: 3.1 scrub ok 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: 4.1c scrub starts 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: 4.1c scrub ok 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: 3.1 scrub starts 2024-03-24T11:42:35.288 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:35 smithi179 ceph-mon[37907]: 3.1 scrub ok 2024-03-24T11:42:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:42:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: 4.1c scrub starts 2024-03-24T11:42:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: 4.1c scrub ok 2024-03-24T11:42:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: osdmap e506: 8 total, 8 up, 8 in 2024-03-24T11:42:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:35.652 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:35.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: 3.1 scrub starts 2024-03-24T11:42:35.653 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:35 smithi181 ceph-mon[33998]: 3.1 scrub ok 2024-03-24T11:42:36.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:42:36.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[29608]: 5.2 deep-scrub starts 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[29608]: 5.2 deep-scrub ok 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[29608]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[29608]: 3.17 scrub starts 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[29608]: 3.17 scrub ok 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[37907]: 5.2 deep-scrub starts 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[37907]: 5.2 deep-scrub ok 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[37907]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T11:42:36.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[37907]: 3.17 scrub starts 2024-03-24T11:42:36.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:36 smithi179 ceph-mon[37907]: 3.17 scrub ok 2024-03-24T11:42:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:36 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:42:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:36 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:42:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:36 smithi181 ceph-mon[33998]: 5.2 deep-scrub starts 2024-03-24T11:42:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:36 smithi181 ceph-mon[33998]: 5.2 deep-scrub ok 2024-03-24T11:42:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:36 smithi181 ceph-mon[33998]: osdmap e507: 8 total, 8 up, 8 in 2024-03-24T11:42:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:36 smithi181 ceph-mon[33998]: 3.17 scrub starts 2024-03-24T11:42:36.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:36 smithi181 ceph-mon[33998]: 3.17 scrub ok 2024-03-24T11:42:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 3.14 deep-scrub starts 2024-03-24T11:42:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 3.14 deep-scrub ok 2024-03-24T11:42:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 1.6 scrub starts 2024-03-24T11:42:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 1.6 scrub ok 2024-03-24T11:42:37.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 1.8 scrub starts 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 1.8 scrub ok 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 4.b scrub starts 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 4.b scrub ok 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 3.1e scrub starts 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[29608]: 3.1e scrub ok 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 3.14 deep-scrub starts 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 3.14 deep-scrub ok 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 1.6 scrub starts 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 1.6 scrub ok 2024-03-24T11:42:37.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 1.8 scrub starts 2024-03-24T11:42:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 1.8 scrub ok 2024-03-24T11:42:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 4.b scrub starts 2024-03-24T11:42:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 4.b scrub ok 2024-03-24T11:42:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 3.1e scrub starts 2024-03-24T11:42:37.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:37 smithi179 ceph-mon[37907]: 3.1e scrub ok 2024-03-24T11:42:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 3.14 deep-scrub starts 2024-03-24T11:42:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 3.14 deep-scrub ok 2024-03-24T11:42:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 1.6 scrub starts 2024-03-24T11:42:37.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 1.6 scrub ok 2024-03-24T11:42:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 1.8 scrub starts 2024-03-24T11:42:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 1.8 scrub ok 2024-03-24T11:42:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 4.b scrub starts 2024-03-24T11:42:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 4.b scrub ok 2024-03-24T11:42:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 3.1e scrub starts 2024-03-24T11:42:37.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:37 smithi181 ceph-mon[33998]: 3.1e scrub ok 2024-03-24T11:42:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[29608]: 5.1 scrub starts 2024-03-24T11:42:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[29608]: 5.1 scrub ok 2024-03-24T11:42:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[29608]: 5.10 scrub starts 2024-03-24T11:42:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[29608]: 5.10 scrub ok 2024-03-24T11:42:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[29608]: 6.16 scrub starts 2024-03-24T11:42:38.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[29608]: 6.1 scrub starts 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[29608]: 6.1 scrub ok 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[37907]: 5.1 scrub starts 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[37907]: 5.1 scrub ok 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[37907]: 5.10 scrub starts 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[37907]: 5.10 scrub ok 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[37907]: 6.16 scrub starts 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[37907]: 6.1 scrub starts 2024-03-24T11:42:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:38 smithi179 ceph-mon[37907]: 6.1 scrub ok 2024-03-24T11:42:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:38 smithi181 ceph-mon[33998]: 5.1 scrub starts 2024-03-24T11:42:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:38 smithi181 ceph-mon[33998]: 5.1 scrub ok 2024-03-24T11:42:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:38 smithi181 ceph-mon[33998]: 5.10 scrub starts 2024-03-24T11:42:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:38 smithi181 ceph-mon[33998]: 5.10 scrub ok 2024-03-24T11:42:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:38 smithi181 ceph-mon[33998]: 6.16 scrub starts 2024-03-24T11:42:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:38 smithi181 ceph-mon[33998]: 6.1 scrub starts 2024-03-24T11:42:38.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:38 smithi181 ceph-mon[33998]: 6.1 scrub ok 2024-03-24T11:42:39.315 INFO:teuthology.orchestra.run.smithi179.stdout:55834575168 2024-03-24T11:42:39.316 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:42:39.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[29608]: 6.16 scrub ok 2024-03-24T11:42:39.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[29608]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:42:39.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[29608]: 3.12 deep-scrub starts 2024-03-24T11:42:39.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[29608]: 3.12 deep-scrub ok 2024-03-24T11:42:39.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[29608]: 5.14 scrub starts 2024-03-24T11:42:39.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[29608]: 5.14 scrub ok 2024-03-24T11:42:39.671 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[29608]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T11:42:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[37907]: 6.16 scrub ok 2024-03-24T11:42:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[37907]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:42:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[37907]: 3.12 deep-scrub starts 2024-03-24T11:42:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[37907]: 3.12 deep-scrub ok 2024-03-24T11:42:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[37907]: 5.14 scrub starts 2024-03-24T11:42:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[37907]: 5.14 scrub ok 2024-03-24T11:42:39.672 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:39 smithi179 ceph-mon[37907]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T11:42:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:39 smithi181 ceph-mon[33998]: 6.16 scrub ok 2024-03-24T11:42:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:39 smithi181 ceph-mon[33998]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T11:42:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:39 smithi181 ceph-mon[33998]: 3.12 deep-scrub starts 2024-03-24T11:42:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:39 smithi181 ceph-mon[33998]: 3.12 deep-scrub ok 2024-03-24T11:42:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:39 smithi181 ceph-mon[33998]: 5.14 scrub starts 2024-03-24T11:42:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:39 smithi181 ceph-mon[33998]: 5.14 scrub ok 2024-03-24T11:42:39.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:39 smithi181 ceph-mon[33998]: osdmap e508: 8 total, 8 up, 8 in 2024-03-24T11:42:40.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:40 smithi179 ceph-mon[29608]: 4.15 scrub starts 2024-03-24T11:42:40.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:40 smithi179 ceph-mon[29608]: 4.15 scrub ok 2024-03-24T11:42:40.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:40 smithi179 ceph-mon[37907]: 4.15 scrub starts 2024-03-24T11:42:40.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:40 smithi179 ceph-mon[37907]: 4.15 scrub ok 2024-03-24T11:42:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:40 smithi181 ceph-mon[33998]: 4.15 scrub starts 2024-03-24T11:42:40.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:40 smithi181 ceph-mon[33998]: 4.15 scrub ok 2024-03-24T11:42:41.479 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[37907]: 5.18 scrub starts 2024-03-24T11:42:41.479 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[37907]: 5.18 scrub ok 2024-03-24T11:42:41.479 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[37907]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:41.479 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[37907]: 5.0 scrub starts 2024-03-24T11:42:41.479 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[37907]: 5.0 scrub ok 2024-03-24T11:42:41.480 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[29608]: 5.18 scrub starts 2024-03-24T11:42:41.480 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[29608]: 5.18 scrub ok 2024-03-24T11:42:41.480 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[29608]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:41.480 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[29608]: 5.0 scrub starts 2024-03-24T11:42:41.480 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:41 smithi179 ceph-mon[29608]: 5.0 scrub ok 2024-03-24T11:42:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:41 smithi181 ceph-mon[33998]: 5.18 scrub starts 2024-03-24T11:42:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:41 smithi181 ceph-mon[33998]: 5.18 scrub ok 2024-03-24T11:42:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:41 smithi181 ceph-mon[33998]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-03-24T11:42:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:41 smithi181 ceph-mon[33998]: 5.0 scrub starts 2024-03-24T11:42:41.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:41 smithi181 ceph-mon[33998]: 5.0 scrub ok 2024-03-24T11:42:42.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:42 smithi179 ceph-mon[29608]: 6.e deep-scrub starts 2024-03-24T11:42:42.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:42 smithi179 ceph-mon[29608]: 6.e deep-scrub ok 2024-03-24T11:42:42.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:42 smithi179 ceph-mon[37907]: 6.e deep-scrub starts 2024-03-24T11:42:42.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:42 smithi179 ceph-mon[37907]: 6.e deep-scrub ok 2024-03-24T11:42:42.651 INFO:teuthology.orchestra.run.smithi179.stdout:133143986481 2024-03-24T11:42:42.651 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T11:42:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:42 smithi181 ceph-mon[33998]: 6.e deep-scrub starts 2024-03-24T11:42:42.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:42 smithi181 ceph-mon[33998]: 6.e deep-scrub ok 2024-03-24T11:42:43.322 INFO:teuthology.orchestra.run.smithi179.stdout:214748365094 2024-03-24T11:42:43.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T11:42:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:42:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:42:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[29608]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T11:42:43.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[29608]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-24T11:42:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:42:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:42:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[37907]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T11:42:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:43 smithi179 ceph-mon[37907]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-24T11:42:43.738 INFO:teuthology.orchestra.run.smithi179.stdout:34359738694 2024-03-24T11:42:43.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:42:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:43 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:42:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:43 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:42:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:43 smithi181 ceph-mon[33998]: osdmap e509: 8 total, 8 up, 8 in 2024-03-24T11:42:43.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:43 smithi181 ceph-mon[33998]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-24T11:42:44.468 INFO:teuthology.orchestra.run.smithi179.stdout:111669150004 2024-03-24T11:42:44.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T11:42:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:44 smithi179 ceph-mon[29608]: 3.13 scrub starts 2024-03-24T11:42:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:44 smithi179 ceph-mon[29608]: 3.13 scrub ok 2024-03-24T11:42:44.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:44 smithi179 ceph-mon[29608]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T11:42:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:44 smithi179 ceph-mon[37907]: 3.13 scrub starts 2024-03-24T11:42:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:44 smithi179 ceph-mon[37907]: 3.13 scrub ok 2024-03-24T11:42:44.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:44 smithi179 ceph-mon[37907]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T11:42:44.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:42:44 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:42:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:44 smithi181 ceph-mon[33998]: 3.13 scrub starts 2024-03-24T11:42:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:44 smithi181 ceph-mon[33998]: 3.13 scrub ok 2024-03-24T11:42:44.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:44 smithi181 ceph-mon[33998]: osdmap e510: 8 total, 8 up, 8 in 2024-03-24T11:42:45.362 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:42:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: 3.c deep-scrub starts 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: 3.c deep-scrub ok 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: 4.19 scrub starts 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: 4.19 scrub ok 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: 1.7 deep-scrub starts 2024-03-24T11:42:45.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[29608]: 1.7 deep-scrub ok 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: 3.c deep-scrub starts 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: 3.c deep-scrub ok 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: 4.19 scrub starts 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: 4.19 scrub ok 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: 1.7 deep-scrub starts 2024-03-24T11:42:45.658 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:45 smithi179 ceph-mon[37907]: 1.7 deep-scrub ok 2024-03-24T11:42:45.797 INFO:teuthology.orchestra.run.smithi179.stdout:176093659437 2024-03-24T11:42:45.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T11:42:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: 3.c deep-scrub starts 2024-03-24T11:42:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: 3.c deep-scrub ok 2024-03-24T11:42:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: 4.19 scrub starts 2024-03-24T11:42:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: 4.19 scrub ok 2024-03-24T11:42:45.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: 1.7 deep-scrub starts 2024-03-24T11:42:45.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:45 smithi181 ceph-mon[33998]: 1.7 deep-scrub ok 2024-03-24T11:42:46.657 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:46 smithi179 ceph-mon[29608]: 6.9 scrub starts 2024-03-24T11:42:46.657 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:46 smithi179 ceph-mon[37907]: 6.9 scrub starts 2024-03-24T11:42:46.840 INFO:teuthology.orchestra.run.smithi179.stdout:77309411645 2024-03-24T11:42:46.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:42:46.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:46 smithi181 ceph-mon[33998]: 6.9 scrub starts 2024-03-24T11:42:47.328 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:47.328 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:47.328 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:47.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[29608]: 6.9 scrub ok 2024-03-24T11:42:47.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[29608]: 3.1d scrub starts 2024-03-24T11:42:47.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[29608]: 3.1d scrub ok 2024-03-24T11:42:47.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[29608]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 717 B/s rd, 0 op/s 2024-03-24T11:42:47.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[37907]: 6.9 scrub ok 2024-03-24T11:42:47.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[37907]: 3.1d scrub starts 2024-03-24T11:42:47.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[37907]: 3.1d scrub ok 2024-03-24T11:42:47.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:47 smithi179 ceph-mon[37907]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 717 B/s rd, 0 op/s 2024-03-24T11:42:47.822 INFO:teuthology.orchestra.run.smithi179.stdout:253403070755 2024-03-24T11:42:47.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T11:42:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:47 smithi181 ceph-mon[33998]: 6.9 scrub ok 2024-03-24T11:42:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:47 smithi181 ceph-mon[33998]: 3.1d scrub starts 2024-03-24T11:42:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:47 smithi181 ceph-mon[33998]: 3.1d scrub ok 2024-03-24T11:42:47.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:47 smithi181 ceph-mon[33998]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 717 B/s rd, 0 op/s 2024-03-24T11:42:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[29608]: 3.d scrub starts 2024-03-24T11:42:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[29608]: 3.d scrub ok 2024-03-24T11:42:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[29608]: 3.1b deep-scrub starts 2024-03-24T11:42:48.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[29608]: 3.1b deep-scrub ok 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[29608]: 3.10 scrub starts 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[29608]: 3.10 scrub ok 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[37907]: 3.d scrub starts 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[37907]: 3.d scrub ok 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[37907]: 3.1b deep-scrub starts 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[37907]: 3.1b deep-scrub ok 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[37907]: 3.10 scrub starts 2024-03-24T11:42:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:48 smithi179 ceph-mon[37907]: 3.10 scrub ok 2024-03-24T11:42:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:48 smithi181 ceph-mon[33998]: 3.d scrub starts 2024-03-24T11:42:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:48 smithi181 ceph-mon[33998]: 3.d scrub ok 2024-03-24T11:42:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:48 smithi181 ceph-mon[33998]: 3.1b deep-scrub starts 2024-03-24T11:42:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:48 smithi181 ceph-mon[33998]: 3.1b deep-scrub ok 2024-03-24T11:42:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:48 smithi181 ceph-mon[33998]: 3.10 scrub starts 2024-03-24T11:42:48.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:48 smithi181 ceph-mon[33998]: 3.10 scrub ok 2024-03-24T11:42:49.135 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:49.136 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:49.137 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: 1.9 scrub starts 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: 1.9 scrub ok 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: 3.16 scrub starts 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: 3.16 scrub ok 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: 5.1d scrub starts 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: 5.1d scrub ok 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:49.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: 1.9 scrub starts 2024-03-24T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: 1.9 scrub ok 2024-03-24T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: 3.16 scrub starts 2024-03-24T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: 3.16 scrub ok 2024-03-24T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: 5.1d scrub starts 2024-03-24T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: 5.1d scrub ok 2024-03-24T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:49 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: 1.9 scrub starts 2024-03-24T11:42:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: 1.9 scrub ok 2024-03-24T11:42:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: 3.16 scrub starts 2024-03-24T11:42:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: 3.16 scrub ok 2024-03-24T11:42:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: 5.1d scrub starts 2024-03-24T11:42:49.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: 5.1d scrub ok 2024-03-24T11:42:49.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:49 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[29608]: 4.3 scrub starts 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[29608]: 4.3 scrub ok 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[29608]: 3.4 deep-scrub starts 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[29608]: 3.4 deep-scrub ok 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[37907]: 4.3 scrub starts 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[37907]: 4.3 scrub ok 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[37907]: 3.4 deep-scrub starts 2024-03-24T11:42:50.706 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:50 smithi179 ceph-mon[37907]: 3.4 deep-scrub ok 2024-03-24T11:42:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:50 smithi181 ceph-mon[33998]: 4.3 scrub starts 2024-03-24T11:42:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:50 smithi181 ceph-mon[33998]: 4.3 scrub ok 2024-03-24T11:42:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:50 smithi181 ceph-mon[33998]: 3.4 deep-scrub starts 2024-03-24T11:42:50.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:50 smithi181 ceph-mon[33998]: 3.4 deep-scrub ok 2024-03-24T11:42:51.369 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:51.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:51 smithi179 ceph-mon[29608]: pgmap v1129: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:51.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:51 smithi179 ceph-mon[37907]: pgmap v1129: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:51.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:51 smithi181 ceph-mon[33998]: pgmap v1129: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:42:52.480 INFO:teuthology.orchestra.run.smithi179.stdout:176093659438 2024-03-24T11:42:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:52 smithi179 ceph-mon[29608]: 5.1e scrub starts 2024-03-24T11:42:52.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:52 smithi179 ceph-mon[29608]: 5.1e scrub ok 2024-03-24T11:42:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:52 smithi179 ceph-mon[37907]: 5.1e scrub starts 2024-03-24T11:42:52.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:52 smithi179 ceph-mon[37907]: 5.1e scrub ok 2024-03-24T11:42:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:52 smithi181 ceph-mon[33998]: 5.1e scrub starts 2024-03-24T11:42:52.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:52 smithi181 ceph-mon[33998]: 5.1e scrub ok 2024-03-24T11:42:53.530 INFO:teuthology.orchestra.run.smithi179.stdout:111669150007 2024-03-24T11:42:53.611 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[29608]: 5.e scrub starts 2024-03-24T11:42:53.611 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[29608]: 5.e scrub ok 2024-03-24T11:42:53.611 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2603642228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:42:53.611 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[29608]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:42:53.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[37907]: 5.e scrub starts 2024-03-24T11:42:53.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[37907]: 5.e scrub ok 2024-03-24T11:42:53.611 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2603642228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:42:53.612 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:53 smithi179 ceph-mon[37907]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:42:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:53 smithi181 ceph-mon[33998]: 5.e scrub starts 2024-03-24T11:42:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:53 smithi181 ceph-mon[33998]: 5.e scrub ok 2024-03-24T11:42:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:53 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2603642228' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:42:53.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:53 smithi181 ceph-mon[33998]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:42:54.770 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:54 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1223461487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:42:54.770 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:54 smithi179 ceph-mon[29608]: 6.4 deep-scrub starts 2024-03-24T11:42:54.770 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:54 smithi179 ceph-mon[29608]: 6.4 deep-scrub ok 2024-03-24T11:42:54.771 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:54 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1223461487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:42:54.771 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:54 smithi179 ceph-mon[37907]: 6.4 deep-scrub starts 2024-03-24T11:42:54.771 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:54 smithi179 ceph-mon[37907]: 6.4 deep-scrub ok 2024-03-24T11:42:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:54 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1223461487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:42:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:54 smithi181 ceph-mon[33998]: 6.4 deep-scrub starts 2024-03-24T11:42:54.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:54 smithi181 ceph-mon[33998]: 6.4 deep-scrub ok 2024-03-24T11:42:54.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:42:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:42:55.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:42:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:42:55.834 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:42:55.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:55 smithi181 ceph-mon[33998]: 4.13 deep-scrub starts 2024-03-24T11:42:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:55 smithi181 ceph-mon[33998]: 4.13 deep-scrub ok 2024-03-24T11:42:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:55 smithi181 ceph-mon[33998]: 2.2 scrub starts 2024-03-24T11:42:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:55 smithi181 ceph-mon[33998]: 2.2 scrub ok 2024-03-24T11:42:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:55 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:55.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:55 smithi181 ceph-mon[33998]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[29608]: 4.13 deep-scrub starts 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[29608]: 4.13 deep-scrub ok 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[29608]: 2.2 scrub starts 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[29608]: 2.2 scrub ok 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[29608]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[37907]: 4.13 deep-scrub starts 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[37907]: 4.13 deep-scrub ok 2024-03-24T11:42:55.947 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[37907]: 2.2 scrub starts 2024-03-24T11:42:55.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[37907]: 2.2 scrub ok 2024-03-24T11:42:55.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:42:55.948 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:55 smithi179 ceph-mon[37907]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-03-24T11:42:56.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[37907]: 4.1e scrub starts 2024-03-24T11:42:56.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[37907]: 4.1e scrub ok 2024-03-24T11:42:56.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[37907]: 5.1c scrub starts 2024-03-24T11:42:56.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[37907]: 5.1c scrub ok 2024-03-24T11:42:56.650 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[29608]: 4.1e scrub starts 2024-03-24T11:42:56.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[29608]: 4.1e scrub ok 2024-03-24T11:42:56.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[29608]: 5.1c scrub starts 2024-03-24T11:42:56.651 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:56 smithi179 ceph-mon[29608]: 5.1c scrub ok 2024-03-24T11:42:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:56 smithi181 ceph-mon[33998]: 4.1e scrub starts 2024-03-24T11:42:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:56 smithi181 ceph-mon[33998]: 4.1e scrub ok 2024-03-24T11:42:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:56 smithi181 ceph-mon[33998]: 5.1c scrub starts 2024-03-24T11:42:56.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:56 smithi181 ceph-mon[33998]: 5.1c scrub ok 2024-03-24T11:42:57.145 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150004 got 111669150007 for osd.3 2024-03-24T11:42:57.145 DEBUG:teuthology.parallel:result is None 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[29608]: 5.5 scrub starts 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[29608]: 5.5 scrub ok 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[29608]: 4.9 scrub starts 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[29608]: 4.9 scrub ok 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[29608]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[29608]: 4.a scrub starts 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[29608]: 4.a scrub ok 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[37907]: 5.5 scrub starts 2024-03-24T11:42:57.618 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[37907]: 5.5 scrub ok 2024-03-24T11:42:57.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[37907]: 4.9 scrub starts 2024-03-24T11:42:57.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[37907]: 4.9 scrub ok 2024-03-24T11:42:57.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[37907]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:57.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[37907]: 4.a scrub starts 2024-03-24T11:42:57.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:57 smithi179 ceph-mon[37907]: 4.a scrub ok 2024-03-24T11:42:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:57 smithi181 ceph-mon[33998]: 5.5 scrub starts 2024-03-24T11:42:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:57 smithi181 ceph-mon[33998]: 5.5 scrub ok 2024-03-24T11:42:57.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:57 smithi181 ceph-mon[33998]: 4.9 scrub starts 2024-03-24T11:42:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:57 smithi181 ceph-mon[33998]: 4.9 scrub ok 2024-03-24T11:42:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:57 smithi181 ceph-mon[33998]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T11:42:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:57 smithi181 ceph-mon[33998]: 4.a scrub starts 2024-03-24T11:42:57.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:57 smithi181 ceph-mon[33998]: 4.a scrub ok 2024-03-24T11:42:58.805 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:58 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:42:58.805 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:58 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:42:58.805 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:58 smithi179 ceph-mon[29608]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T11:42:58.805 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:58 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:42:58.805 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:58 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:42:58.805 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:58 smithi179 ceph-mon[37907]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T11:42:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:58 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:42:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:58 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:42:58.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:58 smithi181 ceph-mon[33998]: osdmap e511: 8 total, 8 up, 8 in 2024-03-24T11:42:59.013 INFO:teuthology.orchestra.run.smithi179.stdout:133143986484 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[29608]: 5.4 scrub starts 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[29608]: 5.4 scrub ok 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[29608]: 6.11 scrub starts 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[29608]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[29608]: 6.11 scrub ok 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2019719005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[29608]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T11:42:59.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[37907]: 5.4 scrub starts 2024-03-24T11:42:59.596 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[37907]: 5.4 scrub ok 2024-03-24T11:42:59.596 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[37907]: 6.11 scrub starts 2024-03-24T11:42:59.596 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[37907]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:42:59.596 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[37907]: 6.11 scrub ok 2024-03-24T11:42:59.596 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2019719005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:42:59.596 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:42:59 smithi179 ceph-mon[37907]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T11:42:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:59 smithi181 ceph-mon[33998]: 5.4 scrub starts 2024-03-24T11:42:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:59 smithi181 ceph-mon[33998]: 5.4 scrub ok 2024-03-24T11:42:59.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:59 smithi181 ceph-mon[33998]: 6.11 scrub starts 2024-03-24T11:42:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:59 smithi181 ceph-mon[33998]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:42:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:59 smithi181 ceph-mon[33998]: 6.11 scrub ok 2024-03-24T11:42:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:59 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2019719005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:42:59.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:42:59 smithi181 ceph-mon[33998]: osdmap e512: 8 total, 8 up, 8 in 2024-03-24T11:42:59.921 INFO:teuthology.orchestra.run.smithi179.stdout:34359738697 2024-03-24T11:43:00.359 INFO:teuthology.orchestra.run.smithi179.stdout:55834575172 2024-03-24T11:43:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[37907]: 4.1 scrub starts 2024-03-24T11:43:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[37907]: 4.1 scrub ok 2024-03-24T11:43:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[37907]: 5.a scrub starts 2024-03-24T11:43:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[37907]: 5.a scrub ok 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/139758060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/1722356957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/965471779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[29608]: 4.1 scrub starts 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[29608]: 4.1 scrub ok 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[29608]: 5.a scrub starts 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[29608]: 5.a scrub ok 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/139758060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/1722356957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:43:00.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:00 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/965471779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:43:00.774 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738694 got 34359738697 for osd.0 2024-03-24T11:43:00.774 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:00 smithi181 ceph-mon[33998]: 4.1 scrub starts 2024-03-24T11:43:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:00 smithi181 ceph-mon[33998]: 4.1 scrub ok 2024-03-24T11:43:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:00 smithi181 ceph-mon[33998]: 5.a scrub starts 2024-03-24T11:43:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:00 smithi181 ceph-mon[33998]: 5.a scrub ok 2024-03-24T11:43:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/139758060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:43:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/1722356957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:43:00.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:00 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/965471779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:43:01.166 INFO:teuthology.orchestra.run.smithi179.stdout:77309411647 2024-03-24T11:43:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[37907]: 5.c scrub starts 2024-03-24T11:43:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[37907]: 5.c scrub ok 2024-03-24T11:43:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[37907]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[37907]: 6.12 scrub starts 2024-03-24T11:43:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2233853378' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:43:01.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[29608]: 5.c scrub starts 2024-03-24T11:43:01.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[29608]: 5.c scrub ok 2024-03-24T11:43:01.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[29608]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:01.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[29608]: 6.12 scrub starts 2024-03-24T11:43:01.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:01 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2233853378' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:43:01.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411645 got 77309411647 for osd.2 2024-03-24T11:43:01.781 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:01 smithi181 ceph-mon[33998]: 5.c scrub starts 2024-03-24T11:43:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:01 smithi181 ceph-mon[33998]: 5.c scrub ok 2024-03-24T11:43:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:01 smithi181 ceph-mon[33998]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:01 smithi181 ceph-mon[33998]: 6.12 scrub starts 2024-03-24T11:43:01.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:01 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2233853378' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T11:43:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:02 smithi179 ceph-mon[37907]: 6.12 scrub ok 2024-03-24T11:43:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:02 smithi179 ceph-mon[37907]: 6.1a scrub starts 2024-03-24T11:43:02.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:02 smithi179 ceph-mon[37907]: 6.1a scrub ok 2024-03-24T11:43:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:02 smithi179 ceph-mon[29608]: 6.12 scrub ok 2024-03-24T11:43:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:02 smithi179 ceph-mon[29608]: 6.1a scrub starts 2024-03-24T11:43:02.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:02 smithi179 ceph-mon[29608]: 6.1a scrub ok 2024-03-24T11:43:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:02 smithi181 ceph-mon[33998]: 6.12 scrub ok 2024-03-24T11:43:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:02 smithi181 ceph-mon[33998]: 6.1a scrub starts 2024-03-24T11:43:02.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:02 smithi181 ceph-mon[33998]: 6.1a scrub ok 2024-03-24T11:43:03.362 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659437 got 176093659438 for osd.5 2024-03-24T11:43:03.362 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[37907]: 6.1d scrub starts 2024-03-24T11:43:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[37907]: 6.6 scrub starts 2024-03-24T11:43:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[37907]: 6.6 scrub ok 2024-03-24T11:43:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[37907]: 6.1d scrub ok 2024-03-24T11:43:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[37907]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:43:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[29608]: 6.1d scrub starts 2024-03-24T11:43:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[29608]: 6.6 scrub starts 2024-03-24T11:43:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[29608]: 6.6 scrub ok 2024-03-24T11:43:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[29608]: 6.1d scrub ok 2024-03-24T11:43:03.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:03 smithi179 ceph-mon[29608]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:43:03.796 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986481 got 133143986484 for osd.4 2024-03-24T11:43:03.796 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:03.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:03 smithi181 ceph-mon[33998]: 6.1d scrub starts 2024-03-24T11:43:03.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:03 smithi181 ceph-mon[33998]: 6.6 scrub starts 2024-03-24T11:43:03.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:03 smithi181 ceph-mon[33998]: 6.6 scrub ok 2024-03-24T11:43:03.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:03 smithi181 ceph-mon[33998]: 6.1d scrub ok 2024-03-24T11:43:03.803 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:03 smithi181 ceph-mon[33998]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T11:43:03.887 INFO:teuthology.orchestra.run.smithi179.stdout:214748365098 2024-03-24T11:43:04.454 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365094 got 214748365098 for osd.6 2024-03-24T11:43:04.454 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:04.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:04 smithi179 ceph-mon[29608]: 6.10 scrub starts 2024-03-24T11:43:04.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:04 smithi179 ceph-mon[29608]: 6.10 scrub ok 2024-03-24T11:43:04.673 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:04 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:04.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:04 smithi179 ceph-mon[37907]: 6.10 scrub starts 2024-03-24T11:43:04.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:04 smithi179 ceph-mon[37907]: 6.10 scrub ok 2024-03-24T11:43:04.673 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:04 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:04 smithi181 ceph-mon[33998]: 6.10 scrub starts 2024-03-24T11:43:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:04 smithi181 ceph-mon[33998]: 6.10 scrub ok 2024-03-24T11:43:04.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:04 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:43:04.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:43:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:43:05.083 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575168 got 55834575172 for osd.1 2024-03-24T11:43:05.084 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:05.316 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:05 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:43:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:05.524 INFO:teuthology.orchestra.run.smithi179.stdout:253403070759 2024-03-24T11:43:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:05 smithi179 ceph-mon[29608]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:05.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:05 smithi179 ceph-mon[29608]: pgmap v1138: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:05 smithi179 ceph-mon[37907]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:05 smithi179 ceph-mon[37907]: pgmap v1138: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:05 smithi181 ceph-mon[33998]: from='client.24424 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T11:43:05.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:05 smithi181 ceph-mon[33998]: pgmap v1138: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T11:43:06.203 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070755 got 253403070759 for osd.7 2024-03-24T11:43:06.203 DEBUG:teuthology.parallel:result is None 2024-03-24T11:43:06.203 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T11:43:06.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:43:06.452 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[29608]: 3.15 scrub starts 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[29608]: 3.15 scrub ok 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[29608]: 4.f scrub starts 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[29608]: 4.f scrub ok 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/2015951110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[29608]: 4.7 scrub starts 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[29608]: 4.7 scrub ok 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[37907]: 3.15 scrub starts 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[37907]: 3.15 scrub ok 2024-03-24T11:43:06.808 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[37907]: 4.f scrub starts 2024-03-24T11:43:06.809 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[37907]: 4.f scrub ok 2024-03-24T11:43:06.809 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/2015951110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:43:06.809 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[37907]: 4.7 scrub starts 2024-03-24T11:43:06.809 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:06 smithi179 ceph-mon[37907]: 4.7 scrub ok 2024-03-24T11:43:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:06 smithi181 ceph-mon[33998]: 3.15 scrub starts 2024-03-24T11:43:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:06 smithi181 ceph-mon[33998]: 3.15 scrub ok 2024-03-24T11:43:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:06 smithi181 ceph-mon[33998]: 4.f scrub starts 2024-03-24T11:43:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:06 smithi181 ceph-mon[33998]: 4.f scrub ok 2024-03-24T11:43:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:06 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/2015951110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T11:43:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:06 smithi181 ceph-mon[33998]: 4.7 scrub starts 2024-03-24T11:43:06.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:06 smithi181 ceph-mon[33998]: 4.7 scrub ok 2024-03-24T11:43:07.789 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:07 smithi179 ceph-mon[29608]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:43:07.789 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:07 smithi179 ceph-mon[37907]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:43:07.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:07 smithi181 ceph-mon[33998]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:43:08.359 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:43:08.362 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:43:08.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:08 smithi179 ceph-mon[29608]: 2.0 scrub starts 2024-03-24T11:43:08.680 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:08 smithi179 ceph-mon[29608]: 2.0 scrub ok 2024-03-24T11:43:08.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:08 smithi179 ceph-mon[37907]: 2.0 scrub starts 2024-03-24T11:43:08.680 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:08 smithi179 ceph-mon[37907]: 2.0 scrub ok 2024-03-24T11:43:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:08 smithi181 ceph-mon[33998]: 2.0 scrub starts 2024-03-24T11:43:08.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:08 smithi181 ceph-mon[33998]: 2.0 scrub ok 2024-03-24T11:43:09.075 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":1139,"stamp":"2024-03-24T11:43:06.534660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2233,"num_read_kb":1985,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":992703,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1538,"ondisk_log_size":1538,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6314928,"kb_used_data":11100,"kb_used_omap":12,"kb_used_meta":6303731,"kb_avail":743646288,"statfs":{"total":767960285184,"available":761493798912,"internally_reserved":0,"allocated":11366400,"data_stored":7478346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":6455021105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.039212"},"pg_stats":[{"pgid":"4.c","version":"421'30","reported_seq":105,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577091+0000","last_change":"2024-03-24T11:41:34.247353+0000","last_active":"2024-03-24T11:42:59.577091+0000","last_peered":"2024-03-24T11:42:59.577091+0000","last_clean":"2024-03-24T11:42:59.577091+0000","last_became_active":"2024-03-24T11:41:20.439720+0000","last_became_peered":"2024-03-24T11:41:20.439720+0000","last_unstale":"2024-03-24T11:42:59.577091+0000","last_undegraded":"2024-03-24T11:42:59.577091+0000","last_fullsized":"2024-03-24T11:42:59.577091+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:41:34.247251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:34.247251+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:03.233943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":980,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404035+0000","last_change":"2024-03-24T11:41:59.059226+0000","last_active":"2024-03-24T11:42:59.404035+0000","last_peered":"2024-03-24T11:42:59.404035+0000","last_clean":"2024-03-24T11:42:59.404035+0000","last_became_active":"2024-03-24T11:41:20.435862+0000","last_became_peered":"2024-03-24T11:41:20.435862+0000","last_unstale":"2024-03-24T11:42:59.404035+0000","last_undegraded":"2024-03-24T11:42:59.404035+0000","last_fullsized":"2024-03-24T11:42:59.404035+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:59.059120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:59.059120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:17.697948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"469'11","reported_seq":1336,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405212+0000","last_change":"2024-03-24T11:42:18.337741+0000","last_active":"2024-03-24T11:42:59.405212+0000","last_peered":"2024-03-24T11:42:59.405212+0000","last_clean":"2024-03-24T11:42:59.405212+0000","last_became_active":"2024-03-24T11:41:12.169429+0000","last_became_peered":"2024-03-24T11:41:12.169429+0000","last_unstale":"2024-03-24T11:42:59.405212+0000","last_undegraded":"2024-03-24T11:42:59.405212+0000","last_fullsized":"2024-03-24T11:42:59.405212+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'11","last_scrub_stamp":"2024-03-24T11:42:18.337621+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.337621+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:43.657961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577444+0000","last_change":"2024-03-24T11:42:40.923244+0000","last_active":"2024-03-24T11:42:59.577444+0000","last_peered":"2024-03-24T11:42:59.577444+0000","last_clean":"2024-03-24T11:42:59.577444+0000","last_became_active":"2024-03-24T11:41:20.429395+0000","last_became_peered":"2024-03-24T11:41:20.429395+0000","last_unstale":"2024-03-24T11:42:59.577444+0000","last_undegraded":"2024-03-24T11:42:59.577444+0000","last_fullsized":"2024-03-24T11:42:59.577444+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:40.923126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:40.923126+0000","last_clean_scrub_stamp":"2024-03-24T11:42:40.923126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:52.913433+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":992,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013225+0000","last_change":"2024-03-24T11:42:47.987021+0000","last_active":"2024-03-24T11:43:00.013225+0000","last_peered":"2024-03-24T11:43:00.013225+0000","last_clean":"2024-03-24T11:43:00.013225+0000","last_became_active":"2024-03-24T11:42:23.704216+0000","last_became_peered":"2024-03-24T11:42:23.704216+0000","last_unstale":"2024-03-24T11:43:00.013225+0000","last_undegraded":"2024-03-24T11:43:00.013225+0000","last_fullsized":"2024-03-24T11:43:00.013225+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:47.986921+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:41:16.105950+0000","last_clean_scrub_stamp":"2024-03-24T11:42:47.986921+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.485326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"421'53","reported_seq":111,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577501+0000","last_change":"2024-03-24T11:42:13.131062+0000","last_active":"2024-03-24T11:42:59.577501+0000","last_peered":"2024-03-24T11:42:59.577501+0000","last_clean":"2024-03-24T11:42:59.577501+0000","last_became_active":"2024-03-24T11:41:20.434934+0000","last_became_peered":"2024-03-24T11:41:20.434934+0000","last_unstale":"2024-03-24T11:42:59.577501+0000","last_undegraded":"2024-03-24T11:42:59.577501+0000","last_fullsized":"2024-03-24T11:42:59.577501+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'53","last_scrub_stamp":"2024-03-24T11:42:13.130932+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:42:13.130932+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:41.288616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":1045,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976559+0000","last_change":"2024-03-24T11:42:15.407993+0000","last_active":"2024-03-24T11:42:59.976559+0000","last_peered":"2024-03-24T11:42:59.976559+0000","last_clean":"2024-03-24T11:42:59.976559+0000","last_became_active":"2024-03-24T11:41:20.434125+0000","last_became_peered":"2024-03-24T11:41:20.434125+0000","last_unstale":"2024-03-24T11:42:59.976559+0000","last_undegraded":"2024-03-24T11:42:59.976559+0000","last_fullsized":"2024-03-24T11:42:59.976559+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:42:15.407821+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:42:15.407821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:35.796023+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1013,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.268527+0000","last_change":"2024-03-24T11:43:00.268546+0000","last_active":"2024-03-24T11:43:00.268527+0000","last_peered":"2024-03-24T11:43:00.268527+0000","last_clean":"2024-03-24T11:43:00.268527+0000","last_became_active":"2024-03-24T11:41:20.434444+0000","last_became_peered":"2024-03-24T11:41:20.434444+0000","last_unstale":"2024-03-24T11:43:00.268527+0000","last_undegraded":"2024-03-24T11:43:00.268527+0000","last_fullsized":"2024-03-24T11:43:00.268527+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:00.268472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:18.788516+0000","last_clean_scrub_stamp":"2024-03-24T11:43:00.268472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:09.297578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":771,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405024+0000","last_change":"2024-03-24T11:42:01.492944+0000","last_active":"2024-03-24T11:42:59.405024+0000","last_peered":"2024-03-24T11:42:59.405024+0000","last_clean":"2024-03-24T11:42:59.405024+0000","last_became_active":"2024-03-24T11:41:49.149566+0000","last_became_peered":"2024-03-24T11:41:49.149566+0000","last_unstale":"2024-03-24T11:42:59.405024+0000","last_undegraded":"2024-03-24T11:42:59.405024+0000","last_fullsized":"2024-03-24T11:42:59.405024+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:01.492838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:01.492838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:07.335903+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":1159,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401662+0000","last_change":"2024-03-24T11:42:36.345577+0000","last_active":"2024-03-24T11:42:59.401662+0000","last_peered":"2024-03-24T11:42:59.401662+0000","last_clean":"2024-03-24T11:42:59.401662+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:42:59.401662+0000","last_undegraded":"2024-03-24T11:42:59.401662+0000","last_fullsized":"2024-03-24T11:42:59.401662+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:42:36.345473+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.345473+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:54.986531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":105,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577253+0000","last_change":"2024-03-24T11:42:02.119690+0000","last_active":"2024-03-24T11:42:59.577253+0000","last_peered":"2024-03-24T11:42:59.577253+0000","last_clean":"2024-03-24T11:42:59.577253+0000","last_became_active":"2024-03-24T11:41:20.433184+0000","last_became_peered":"2024-03-24T11:41:20.433184+0000","last_unstale":"2024-03-24T11:42:59.577253+0000","last_undegraded":"2024-03-24T11:42:59.577253+0000","last_fullsized":"2024-03-24T11:42:59.577253+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:42:02.119468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:02.119468+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:24.522485+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576798+0000","last_change":"2024-03-24T11:42:18.038800+0000","last_active":"2024-03-24T11:42:59.576798+0000","last_peered":"2024-03-24T11:42:59.576798+0000","last_clean":"2024-03-24T11:42:59.576798+0000","last_became_active":"2024-03-24T11:41:20.431249+0000","last_became_peered":"2024-03-24T11:41:20.431249+0000","last_unstale":"2024-03-24T11:42:59.576798+0000","last_undegraded":"2024-03-24T11:42:59.576798+0000","last_fullsized":"2024-03-24T11:42:59.576798+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:18.038677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.038677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:28.393071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":997,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402711+0000","last_change":"2024-03-24T11:42:22.254996+0000","last_active":"2024-03-24T11:42:59.402711+0000","last_peered":"2024-03-24T11:42:59.402711+0000","last_clean":"2024-03-24T11:42:59.402711+0000","last_became_active":"2024-03-24T11:41:20.423777+0000","last_became_peered":"2024-03-24T11:41:20.423777+0000","last_unstale":"2024-03-24T11:42:59.402711+0000","last_undegraded":"2024-03-24T11:42:59.402711+0000","last_fullsized":"2024-03-24T11:42:59.402711+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:22.254896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:42:22.254896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:35.361462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":956,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404155+0000","last_change":"2024-03-24T11:42:08.920820+0000","last_active":"2024-03-24T11:42:59.404155+0000","last_peered":"2024-03-24T11:42:59.404155+0000","last_clean":"2024-03-24T11:42:59.404155+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:42:59.404155+0000","last_undegraded":"2024-03-24T11:42:59.404155+0000","last_fullsized":"2024-03-24T11:42:59.404155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:08.920744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:08.920744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:22.522446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"421'46","reported_seq":1153,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014228+0000","last_change":"2024-03-24T11:41:56.169237+0000","last_active":"2024-03-24T11:43:00.014228+0000","last_peered":"2024-03-24T11:43:00.014228+0000","last_clean":"2024-03-24T11:43:00.014228+0000","last_became_active":"2024-03-24T11:41:20.435397+0000","last_became_peered":"2024-03-24T11:41:20.435397+0000","last_unstale":"2024-03-24T11:43:00.014228+0000","last_undegraded":"2024-03-24T11:43:00.014228+0000","last_fullsized":"2024-03-24T11:43:00.014228+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-24T11:41:56.169132+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:41:56.169132+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:04.652590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":989,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403414+0000","last_change":"2024-03-24T11:42:18.843034+0000","last_active":"2024-03-24T11:42:59.403414+0000","last_peered":"2024-03-24T11:42:59.403414+0000","last_clean":"2024-03-24T11:42:59.403414+0000","last_became_active":"2024-03-24T11:41:12.177833+0000","last_became_peered":"2024-03-24T11:41:12.177833+0000","last_unstale":"2024-03-24T11:42:59.403414+0000","last_undegraded":"2024-03-24T11:42:59.403414+0000","last_fullsized":"2024-03-24T11:42:59.403414+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:18.842953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.842953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:27.663545+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"477'10","reported_seq":125,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577341+0000","last_change":"2024-03-24T11:42:51.937833+0000","last_active":"2024-03-24T11:42:59.577341+0000","last_peered":"2024-03-24T11:42:59.577341+0000","last_clean":"2024-03-24T11:42:59.577341+0000","last_became_active":"2024-03-24T11:41:20.443531+0000","last_became_peered":"2024-03-24T11:41:20.443531+0000","last_unstale":"2024-03-24T11:42:59.577341+0000","last_undegraded":"2024-03-24T11:42:59.577341+0000","last_fullsized":"2024-03-24T11:42:59.577341+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'10","last_scrub_stamp":"2024-03-24T11:42:51.937705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:51.937705+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.912013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":973,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402178+0000","last_change":"2024-03-24T11:41:34.156336+0000","last_active":"2024-03-24T11:42:59.402178+0000","last_peered":"2024-03-24T11:42:59.402178+0000","last_clean":"2024-03-24T11:42:59.402178+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:42:59.402178+0000","last_undegraded":"2024-03-24T11:42:59.402178+0000","last_fullsized":"2024-03-24T11:42:59.402178+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:34.156230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:41:34.156230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":1150,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401616+0000","last_change":"2024-03-24T11:42:32.355252+0000","last_active":"2024-03-24T11:42:59.401616+0000","last_peered":"2024-03-24T11:42:59.401616+0000","last_clean":"2024-03-24T11:42:59.401616+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:42:59.401616+0000","last_undegraded":"2024-03-24T11:42:59.401616+0000","last_fullsized":"2024-03-24T11:42:59.401616+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:32.355115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:42:32.355115+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:01.448797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576527+0000","last_change":"2024-03-24T11:42:36.858444+0000","last_active":"2024-03-24T11:42:59.576527+0000","last_peered":"2024-03-24T11:42:59.576527+0000","last_clean":"2024-03-24T11:42:59.576527+0000","last_became_active":"2024-03-24T11:41:20.439996+0000","last_became_peered":"2024-03-24T11:41:20.439996+0000","last_unstale":"2024-03-24T11:42:59.576527+0000","last_undegraded":"2024-03-24T11:42:59.576527+0000","last_fullsized":"2024-03-24T11:42:59.576527+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:36.858349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.858349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.216884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'50","reported_seq":1082,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403454+0000","last_change":"2024-03-24T11:41:33.298242+0000","last_active":"2024-03-24T11:42:59.403454+0000","last_peered":"2024-03-24T11:42:59.403454+0000","last_clean":"2024-03-24T11:42:59.403454+0000","last_became_active":"2024-03-24T11:41:12.178352+0000","last_became_peered":"2024-03-24T11:41:12.178352+0000","last_unstale":"2024-03-24T11:42:59.403454+0000","last_undegraded":"2024-03-24T11:42:59.403454+0000","last_fullsized":"2024-03-24T11:42:59.403454+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'50","last_scrub_stamp":"2024-03-24T11:41:33.298084+0000","last_deep_scrub":"479'50","last_deep_scrub_stamp":"2024-03-24T11:41:33.298084+0000","last_clean_scrub_stamp":"2024-03-24T11:41:33.298084+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:03.215517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":981,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404563+0000","last_change":"2024-03-24T11:41:53.167160+0000","last_active":"2024-03-24T11:42:59.404563+0000","last_peered":"2024-03-24T11:42:59.404563+0000","last_clean":"2024-03-24T11:42:59.404563+0000","last_became_active":"2024-03-24T11:41:12.177324+0000","last_became_peered":"2024-03-24T11:41:12.177324+0000","last_unstale":"2024-03-24T11:42:59.404563+0000","last_undegraded":"2024-03-24T11:42:59.404563+0000","last_fullsized":"2024-03-24T11:42:59.404563+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:53.167086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:41:53.167086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:07.262396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576673+0000","last_change":"2024-03-24T11:42:11.067151+0000","last_active":"2024-03-24T11:42:59.576673+0000","last_peered":"2024-03-24T11:42:59.576673+0000","last_clean":"2024-03-24T11:42:59.576673+0000","last_became_active":"2024-03-24T11:41:20.441367+0000","last_became_peered":"2024-03-24T11:41:20.441367+0000","last_unstale":"2024-03-24T11:42:59.576673+0000","last_undegraded":"2024-03-24T11:42:59.576673+0000","last_fullsized":"2024-03-24T11:42:59.576673+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:11.067050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:42:11.067050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.818522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":618,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577504+0000","last_change":"2024-03-24T11:42:22.990318+0000","last_active":"2024-03-24T11:42:59.577504+0000","last_peered":"2024-03-24T11:42:59.577504+0000","last_clean":"2024-03-24T11:42:59.577504+0000","last_became_active":"2024-03-24T11:41:52.546134+0000","last_became_peered":"2024-03-24T11:41:52.546134+0000","last_unstale":"2024-03-24T11:42:59.577504+0000","last_undegraded":"2024-03-24T11:42:59.577504+0000","last_fullsized":"2024-03-24T11:42:59.577504+0000","mapping_epoch":487,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:22.990174+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_clean_scrub_stamp":"2024-03-24T11:42:22.990174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:30.365210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"477'10","reported_seq":1281,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:04.005331+0000","last_change":"2024-03-24T11:42:40.604076+0000","last_active":"2024-03-24T11:43:04.005331+0000","last_peered":"2024-03-24T11:43:04.005331+0000","last_clean":"2024-03-24T11:43:04.005331+0000","last_became_active":"2024-03-24T11:41:20.422150+0000","last_became_peered":"2024-03-24T11:41:20.422150+0000","last_unstale":"2024-03-24T11:43:04.005331+0000","last_undegraded":"2024-03-24T11:43:04.005331+0000","last_fullsized":"2024-03-24T11:43:04.005331+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'10","last_scrub_stamp":"2024-03-24T11:42:40.603935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:40.603935+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.252161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'42","reported_seq":117,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576659+0000","last_change":"2024-03-24T11:42:59.067343+0000","last_active":"2024-03-24T11:42:59.576659+0000","last_peered":"2024-03-24T11:42:59.576659+0000","last_clean":"2024-03-24T11:42:59.576659+0000","last_became_active":"2024-03-24T11:41:20.436237+0000","last_became_peered":"2024-03-24T11:41:20.436237+0000","last_unstale":"2024-03-24T11:42:59.576659+0000","last_undegraded":"2024-03-24T11:42:59.576659+0000","last_fullsized":"2024-03-24T11:42:59.576659+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'42","last_scrub_stamp":"2024-03-24T11:42:59.067206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:59.067206+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:17.848043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":1043,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013952+0000","last_change":"2024-03-24T11:42:14.441001+0000","last_active":"2024-03-24T11:43:00.013952+0000","last_peered":"2024-03-24T11:43:00.013952+0000","last_clean":"2024-03-24T11:43:00.013952+0000","last_became_active":"2024-03-24T11:41:20.432418+0000","last_became_peered":"2024-03-24T11:41:20.432418+0000","last_unstale":"2024-03-24T11:43:00.013952+0000","last_undegraded":"2024-03-24T11:43:00.013952+0000","last_fullsized":"2024-03-24T11:43:00.013952+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:42:14.440869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:14.440869+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:18.772643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":550,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013496+0000","last_change":"2024-03-24T11:42:23.706157+0000","last_active":"2024-03-24T11:43:00.013496+0000","last_peered":"2024-03-24T11:43:00.013496+0000","last_clean":"2024-03-24T11:43:00.013496+0000","last_became_active":"2024-03-24T11:42:23.704839+0000","last_became_peered":"2024-03-24T11:42:23.704839+0000","last_unstale":"2024-03-24T11:43:00.013496+0000","last_undegraded":"2024-03-24T11:43:00.013496+0000","last_fullsized":"2024-03-24T11:43:00.013496+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:13.448157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:13.448157+0000","last_clean_scrub_stamp":"2024-03-24T11:42:13.448157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:39.386341+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1192,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404009+0000","last_change":"2024-03-24T11:41:43.186226+0000","last_active":"2024-03-24T11:42:59.404009+0000","last_peered":"2024-03-24T11:42:59.404009+0000","last_clean":"2024-03-24T11:42:59.404009+0000","last_became_active":"2024-03-24T11:41:15.157694+0000","last_became_peered":"2024-03-24T11:41:15.157694+0000","last_unstale":"2024-03-24T11:42:59.404009+0000","last_undegraded":"2024-03-24T11:42:59.404009+0000","last_fullsized":"2024-03-24T11:42:59.404009+0000","mapping_epoch":470,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:41:43.186127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:41:43.186127+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:12.002602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":1293,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013283+0000","last_change":"2024-03-24T11:42:29.519698+0000","last_active":"2024-03-24T11:43:00.013283+0000","last_peered":"2024-03-24T11:43:00.013283+0000","last_clean":"2024-03-24T11:43:00.013283+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:43:00.013283+0000","last_undegraded":"2024-03-24T11:43:00.013283+0000","last_fullsized":"2024-03-24T11:43:00.013283+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:42:29.519427+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:41:03.262723+0000","last_clean_scrub_stamp":"2024-03-24T11:42:29.519427+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:40.460395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'30","reported_seq":1079,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013710+0000","last_change":"2024-03-24T11:41:43.021116+0000","last_active":"2024-03-24T11:43:00.013710+0000","last_peered":"2024-03-24T11:43:00.013710+0000","last_clean":"2024-03-24T11:43:00.013710+0000","last_became_active":"2024-03-24T11:41:20.438781+0000","last_became_peered":"2024-03-24T11:41:20.438781+0000","last_unstale":"2024-03-24T11:43:00.013710+0000","last_undegraded":"2024-03-24T11:43:00.013710+0000","last_fullsized":"2024-03-24T11:43:00.013710+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:41:43.021020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:43.021020+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:11.038809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":142,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976841+0000","last_change":"2024-03-24T11:42:28.313341+0000","last_active":"2024-03-24T11:42:59.976841+0000","last_peered":"2024-03-24T11:42:59.976841+0000","last_clean":"2024-03-24T11:42:59.976841+0000","last_became_active":"2024-03-24T11:40:35.291074+0000","last_became_peered":"2024-03-24T11:40:35.291074+0000","last_unstale":"2024-03-24T11:42:59.976841+0000","last_undegraded":"2024-03-24T11:42:59.976841+0000","last_fullsized":"2024-03-24T11:42:59.976841+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:28.313240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:11.388851+0000","last_clean_scrub_stamp":"2024-03-24T11:42:28.313240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:36.809012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":983,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012806+0000","last_change":"2024-03-24T11:42:15.523777+0000","last_active":"2024-03-24T11:43:00.012806+0000","last_peered":"2024-03-24T11:43:00.012806+0000","last_clean":"2024-03-24T11:43:00.012806+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:43:00.012806+0000","last_undegraded":"2024-03-24T11:43:00.012806+0000","last_fullsized":"2024-03-24T11:43:00.012806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:42:15.523653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:15.523653+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:21.046051+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1229,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977157+0000","last_change":"2024-03-24T11:42:45.335848+0000","last_active":"2024-03-24T11:42:59.977157+0000","last_peered":"2024-03-24T11:42:59.977157+0000","last_clean":"2024-03-24T11:42:59.977157+0000","last_became_active":"2024-03-24T11:41:44.506953+0000","last_became_peered":"2024-03-24T11:41:44.506953+0000","last_unstale":"2024-03-24T11:42:59.977157+0000","last_undegraded":"2024-03-24T11:42:59.977157+0000","last_fullsized":"2024-03-24T11:42:59.977157+0000","mapping_epoch":482,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:45.335735+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:42:45.335735+0000","last_clean_scrub_stamp":"2024-03-24T11:42:45.335735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.080281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1018,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976945+0000","last_change":"2024-03-24T11:42:35.366426+0000","last_active":"2024-03-24T11:42:59.976945+0000","last_peered":"2024-03-24T11:42:59.976945+0000","last_clean":"2024-03-24T11:42:59.976945+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:42:59.976945+0000","last_undegraded":"2024-03-24T11:42:59.976945+0000","last_fullsized":"2024-03-24T11:42:59.976945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:35.366331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:35.366331+0000","last_clean_scrub_stamp":"2024-03-24T11:42:35.366331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.513525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'39","reported_seq":1112,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012989+0000","last_change":"2024-03-24T11:42:48.683947+0000","last_active":"2024-03-24T11:43:00.012989+0000","last_peered":"2024-03-24T11:43:00.012989+0000","last_clean":"2024-03-24T11:43:00.012989+0000","last_became_active":"2024-03-24T11:41:12.177000+0000","last_became_peered":"2024-03-24T11:41:12.177000+0000","last_unstale":"2024-03-24T11:43:00.012989+0000","last_undegraded":"2024-03-24T11:43:00.012989+0000","last_fullsized":"2024-03-24T11:43:00.012989+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:42:48.683793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:48.683793+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:10.986638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1010,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013716+0000","last_change":"2024-03-24T11:42:50.129608+0000","last_active":"2024-03-24T11:43:00.013716+0000","last_peered":"2024-03-24T11:43:00.013716+0000","last_clean":"2024-03-24T11:43:00.013716+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:43:00.013716+0000","last_undegraded":"2024-03-24T11:43:00.013716+0000","last_fullsized":"2024-03-24T11:43:00.013716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:50.129525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:50.129525+0000","last_clean_scrub_stamp":"2024-03-24T11:42:50.129525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:17.910565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":990,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012875+0000","last_change":"2024-03-24T11:42:38.080747+0000","last_active":"2024-03-24T11:43:00.012875+0000","last_peered":"2024-03-24T11:43:00.012875+0000","last_clean":"2024-03-24T11:43:00.012875+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:43:00.012875+0000","last_undegraded":"2024-03-24T11:43:00.012875+0000","last_fullsized":"2024-03-24T11:43:00.012875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:38.080629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:38.080629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:43.603608+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1195,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977166+0000","last_change":"2024-03-24T11:42:36.319294+0000","last_active":"2024-03-24T11:42:59.977166+0000","last_peered":"2024-03-24T11:42:59.977166+0000","last_clean":"2024-03-24T11:42:59.977166+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:42:59.977166+0000","last_undegraded":"2024-03-24T11:42:59.977166+0000","last_fullsized":"2024-03-24T11:42:59.977166+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:36.319154+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.319154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:54.924573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":986,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402780+0000","last_change":"2024-03-24T11:41:47.050109+0000","last_active":"2024-03-24T11:42:59.402780+0000","last_peered":"2024-03-24T11:42:59.402780+0000","last_clean":"2024-03-24T11:42:59.402780+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:42:59.402780+0000","last_undegraded":"2024-03-24T11:42:59.402780+0000","last_fullsized":"2024-03-24T11:42:59.402780+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:47.049971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:41:47.049971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:07.661474+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'36","reported_seq":553,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402292+0000","last_change":"2024-03-24T11:41:42.007388+0000","last_active":"2024-03-24T11:42:59.402292+0000","last_peered":"2024-03-24T11:42:59.402292+0000","last_clean":"2024-03-24T11:42:59.402292+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:42:59.402292+0000","last_undegraded":"2024-03-24T11:42:59.402292+0000","last_fullsized":"2024-03-24T11:42:59.402292+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'36","last_scrub_stamp":"2024-03-24T11:41:42.007263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:42.007263+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:09.419440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1263,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:01.757405+0000","last_change":"2024-03-24T11:42:23.698308+0000","last_active":"2024-03-24T11:43:01.757405+0000","last_peered":"2024-03-24T11:43:01.757405+0000","last_clean":"2024-03-24T11:43:01.757405+0000","last_became_active":"2024-03-24T11:42:23.696556+0000","last_became_peered":"2024-03-24T11:42:23.696556+0000","last_unstale":"2024-03-24T11:43:01.757405+0000","last_undegraded":"2024-03-24T11:43:01.757405+0000","last_fullsized":"2024-03-24T11:43:01.757405+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:41:48.060201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:41:48.060201+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":670,"num_read_kb":665,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1017,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012709+0000","last_change":"2024-03-24T11:42:34.573788+0000","last_active":"2024-03-24T11:43:00.012709+0000","last_peered":"2024-03-24T11:43:00.012709+0000","last_clean":"2024-03-24T11:43:00.012709+0000","last_became_active":"2024-03-24T11:41:20.423214+0000","last_became_peered":"2024-03-24T11:41:20.423214+0000","last_unstale":"2024-03-24T11:43:00.012709+0000","last_undegraded":"2024-03-24T11:43:00.012709+0000","last_fullsized":"2024-03-24T11:43:00.012709+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:34.573666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:34.573666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:38.297826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":998,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013628+0000","last_change":"2024-03-24T11:42:54.144105+0000","last_active":"2024-03-24T11:43:00.013628+0000","last_peered":"2024-03-24T11:43:00.013628+0000","last_clean":"2024-03-24T11:43:00.013628+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:43:00.013628+0000","last_undegraded":"2024-03-24T11:43:00.013628+0000","last_fullsized":"2024-03-24T11:43:00.013628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:54.144022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:54.144022+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.144022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:07.489551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1217,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013401+0000","last_change":"2024-03-24T11:41:53.097361+0000","last_active":"2024-03-24T11:43:00.013401+0000","last_peered":"2024-03-24T11:43:00.013401+0000","last_clean":"2024-03-24T11:43:00.013401+0000","last_became_active":"2024-03-24T11:41:52.545645+0000","last_became_peered":"2024-03-24T11:41:52.545645+0000","last_unstale":"2024-03-24T11:43:00.013401+0000","last_undegraded":"2024-03-24T11:43:00.013401+0000","last_fullsized":"2024-03-24T11:43:00.013401+0000","mapping_epoch":487,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:41:53.097276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:41:53.097276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:14.580832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":558,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013670+0000","last_change":"2024-03-24T11:42:57.897663+0000","last_active":"2024-03-24T11:43:00.013670+0000","last_peered":"2024-03-24T11:43:00.013670+0000","last_clean":"2024-03-24T11:43:00.013670+0000","last_became_active":"2024-03-24T11:42:23.704051+0000","last_became_peered":"2024-03-24T11:42:23.704051+0000","last_unstale":"2024-03-24T11:43:00.013670+0000","last_undegraded":"2024-03-24T11:43:00.013670+0000","last_fullsized":"2024-03-24T11:43:00.013670+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:57.897437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:42:57.897437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.930802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"477'52","reported_seq":1466,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977043+0000","last_change":"2024-03-24T11:42:10.342514+0000","last_active":"2024-03-24T11:42:59.977043+0000","last_peered":"2024-03-24T11:42:59.977043+0000","last_clean":"2024-03-24T11:42:59.977043+0000","last_became_active":"2024-03-24T11:41:20.444994+0000","last_became_peered":"2024-03-24T11:41:20.444994+0000","last_unstale":"2024-03-24T11:42:59.977043+0000","last_undegraded":"2024-03-24T11:42:59.977043+0000","last_fullsized":"2024-03-24T11:42:59.977043+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'52","last_scrub_stamp":"2024-03-24T11:42:10.342346+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.342346+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:40.038096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":981,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403821+0000","last_change":"2024-03-24T11:41:41.160247+0000","last_active":"2024-03-24T11:42:59.403821+0000","last_peered":"2024-03-24T11:42:59.403821+0000","last_clean":"2024-03-24T11:42:59.403821+0000","last_became_active":"2024-03-24T11:39:28.584724+0000","last_became_peered":"2024-03-24T11:39:28.584724+0000","last_unstale":"2024-03-24T11:42:59.403821+0000","last_undegraded":"2024-03-24T11:42:59.403821+0000","last_fullsized":"2024-03-24T11:42:59.403821+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:41.160141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:41.160141+0000","last_clean_scrub_stamp":"2024-03-24T11:41:41.160141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":968,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013016+0000","last_change":"2024-03-24T11:42:30.879238+0000","last_active":"2024-03-24T11:43:00.013016+0000","last_peered":"2024-03-24T11:43:00.013016+0000","last_clean":"2024-03-24T11:43:00.013016+0000","last_became_active":"2024-03-24T11:42:23.700644+0000","last_became_peered":"2024-03-24T11:42:23.700644+0000","last_unstale":"2024-03-24T11:43:00.013016+0000","last_undegraded":"2024-03-24T11:43:00.013016+0000","last_fullsized":"2024-03-24T11:43:00.013016+0000","mapping_epoch":502,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:30.879136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:42:30.879136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:41.591889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1010,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402422+0000","last_change":"2024-03-24T11:42:31.531147+0000","last_active":"2024-03-24T11:42:59.402422+0000","last_peered":"2024-03-24T11:42:59.402422+0000","last_clean":"2024-03-24T11:42:59.402422+0000","last_became_active":"2024-03-24T11:41:20.432083+0000","last_became_peered":"2024-03-24T11:41:20.432083+0000","last_unstale":"2024-03-24T11:42:59.402422+0000","last_undegraded":"2024-03-24T11:42:59.402422+0000","last_fullsized":"2024-03-24T11:42:59.402422+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:31.531021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:42:31.531021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:35.817363+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1030,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404860+0000","last_change":"2024-03-24T11:42:27.325301+0000","last_active":"2024-03-24T11:42:59.404860+0000","last_peered":"2024-03-24T11:42:59.404860+0000","last_clean":"2024-03-24T11:42:59.404860+0000","last_became_active":"2024-03-24T11:41:12.170158+0000","last_became_peered":"2024-03-24T11:41:12.170158+0000","last_unstale":"2024-03-24T11:42:59.404860+0000","last_undegraded":"2024-03-24T11:42:59.404860+0000","last_fullsized":"2024-03-24T11:42:59.404860+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:27.325221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.325221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.090749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"449'26","reported_seq":1032,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405224+0000","last_change":"2024-03-24T11:41:45.320795+0000","last_active":"2024-03-24T11:42:59.405224+0000","last_peered":"2024-03-24T11:42:59.405224+0000","last_clean":"2024-03-24T11:42:59.405224+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:42:59.405224+0000","last_undegraded":"2024-03-24T11:42:59.405224+0000","last_fullsized":"2024-03-24T11:42:59.405224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'26","last_scrub_stamp":"2024-03-24T11:41:45.320696+0000","last_deep_scrub":"449'26","last_deep_scrub_stamp":"2024-03-24T11:40:22.408163+0000","last_clean_scrub_stamp":"2024-03-24T11:41:45.320696+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:00.112245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1035,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404295+0000","last_change":"2024-03-24T11:42:32.384269+0000","last_active":"2024-03-24T11:42:59.404295+0000","last_peered":"2024-03-24T11:42:59.404295+0000","last_clean":"2024-03-24T11:42:59.404295+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:42:59.404295+0000","last_undegraded":"2024-03-24T11:42:59.404295+0000","last_fullsized":"2024-03-24T11:42:59.404295+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:32.384181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:21.288390+0000","last_clean_scrub_stamp":"2024-03-24T11:42:32.384181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:47.116360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1011,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013165+0000","last_change":"2024-03-24T11:42:22.106589+0000","last_active":"2024-03-24T11:43:00.013165+0000","last_peered":"2024-03-24T11:43:00.013165+0000","last_clean":"2024-03-24T11:43:00.013165+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:43:00.013165+0000","last_undegraded":"2024-03-24T11:43:00.013165+0000","last_fullsized":"2024-03-24T11:43:00.013165+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:22.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:08.778916+0000","last_clean_scrub_stamp":"2024-03-24T11:42:22.106466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:27.333873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":550,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013983+0000","last_change":"2024-03-24T11:42:23.702546+0000","last_active":"2024-03-24T11:43:00.013983+0000","last_peered":"2024-03-24T11:43:00.013983+0000","last_clean":"2024-03-24T11:43:00.013983+0000","last_became_active":"2024-03-24T11:42:23.701790+0000","last_became_peered":"2024-03-24T11:42:23.701790+0000","last_unstale":"2024-03-24T11:43:00.013983+0000","last_undegraded":"2024-03-24T11:43:00.013983+0000","last_fullsized":"2024-03-24T11:43:00.013983+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:21.466092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:42:21.466092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.644913+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":914,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013241+0000","last_change":"2024-03-24T11:42:23.700364+0000","last_active":"2024-03-24T11:43:00.013241+0000","last_peered":"2024-03-24T11:43:00.013241+0000","last_clean":"2024-03-24T11:43:00.013241+0000","last_became_active":"2024-03-24T11:42:23.700004+0000","last_became_peered":"2024-03-24T11:42:23.700004+0000","last_unstale":"2024-03-24T11:43:00.013241+0000","last_undegraded":"2024-03-24T11:43:00.013241+0000","last_fullsized":"2024-03-24T11:43:00.013241+0000","mapping_epoch":502,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:21.495309+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:42:21.495309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:44.063958+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1006,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013359+0000","last_change":"2024-03-24T11:42:55.714126+0000","last_active":"2024-03-24T11:43:00.013359+0000","last_peered":"2024-03-24T11:43:00.013359+0000","last_clean":"2024-03-24T11:43:00.013359+0000","last_became_active":"2024-03-24T11:41:20.424238+0000","last_became_peered":"2024-03-24T11:41:20.424238+0000","last_unstale":"2024-03-24T11:43:00.013359+0000","last_undegraded":"2024-03-24T11:43:00.013359+0000","last_fullsized":"2024-03-24T11:43:00.013359+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:55.714040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:37.333168+0000","last_clean_scrub_stamp":"2024-03-24T11:42:55.714040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:00.006863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"421'53","reported_seq":1400,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013784+0000","last_change":"2024-03-24T11:42:25.133503+0000","last_active":"2024-03-24T11:43:00.013784+0000","last_peered":"2024-03-24T11:43:00.013784+0000","last_clean":"2024-03-24T11:43:00.013784+0000","last_became_active":"2024-03-24T11:37:33.888447+0000","last_became_peered":"2024-03-24T11:37:33.888447+0000","last_unstale":"2024-03-24T11:43:00.013784+0000","last_undegraded":"2024-03-24T11:43:00.013784+0000","last_fullsized":"2024-03-24T11:43:00.013784+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'53","last_scrub_stamp":"2024-03-24T11:42:25.133396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:25.133396+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.907131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577033+0000","last_change":"2024-03-24T11:42:09.057007+0000","last_active":"2024-03-24T11:42:59.577033+0000","last_peered":"2024-03-24T11:42:59.577033+0000","last_clean":"2024-03-24T11:42:59.577033+0000","last_became_active":"2024-03-24T11:41:20.442409+0000","last_became_peered":"2024-03-24T11:41:20.442409+0000","last_unstale":"2024-03-24T11:42:59.577033+0000","last_undegraded":"2024-03-24T11:42:59.577033+0000","last_fullsized":"2024-03-24T11:42:59.577033+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:09.056911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_clean_scrub_stamp":"2024-03-24T11:42:09.056911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:33.861135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1013,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401808+0000","last_change":"2024-03-24T11:42:54.270181+0000","last_active":"2024-03-24T11:42:59.401808+0000","last_peered":"2024-03-24T11:42:59.401808+0000","last_clean":"2024-03-24T11:42:59.401808+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:42:59.401808+0000","last_undegraded":"2024-03-24T11:42:59.401808+0000","last_fullsized":"2024-03-24T11:42:59.401808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:42:54.270078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.270078+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:05.891492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":989,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:02.475651+0000","last_change":"2024-03-24T11:43:02.475692+0000","last_active":"2024-03-24T11:43:02.475651+0000","last_peered":"2024-03-24T11:43:02.475651+0000","last_clean":"2024-03-24T11:43:02.475651+0000","last_became_active":"2024-03-24T11:41:20.437191+0000","last_became_peered":"2024-03-24T11:41:20.437191+0000","last_unstale":"2024-03-24T11:43:02.475651+0000","last_undegraded":"2024-03-24T11:43:02.475651+0000","last_fullsized":"2024-03-24T11:43:02.475651+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:02.475564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:02.475564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:31.938527+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1181,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401540+0000","last_change":"2024-03-24T11:42:10.200089+0000","last_active":"2024-03-24T11:42:59.401540+0000","last_peered":"2024-03-24T11:42:59.401540+0000","last_clean":"2024-03-24T11:42:59.401540+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:42:59.401540+0000","last_undegraded":"2024-03-24T11:42:59.401540+0000","last_fullsized":"2024-03-24T11:42:59.401540+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:10.200004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.200004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.825400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":1063,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013461+0000","last_change":"2024-03-24T11:42:36.575963+0000","last_active":"2024-03-24T11:43:00.013461+0000","last_peered":"2024-03-24T11:43:00.013461+0000","last_clean":"2024-03-24T11:43:00.013461+0000","last_became_active":"2024-03-24T11:41:20.431968+0000","last_became_peered":"2024-03-24T11:41:20.431968+0000","last_unstale":"2024-03-24T11:43:00.013461+0000","last_undegraded":"2024-03-24T11:43:00.013461+0000","last_fullsized":"2024-03-24T11:43:00.013461+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:36.575864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.575864+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.461109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"510'2","reported_seq":1008,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402237+0000","last_change":"2024-03-24T11:42:44.413123+0000","last_active":"2024-03-24T11:42:59.402237+0000","last_peered":"2024-03-24T11:42:59.402237+0000","last_clean":"2024-03-24T11:42:59.402237+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:42:59.402237+0000","last_undegraded":"2024-03-24T11:42:59.402237+0000","last_fullsized":"2024-03-24T11:42:59.402237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'2","last_scrub_stamp":"2024-03-24T11:42:44.412955+0000","last_deep_scrub":"510'2","last_deep_scrub_stamp":"2024-03-24T11:42:44.412955+0000","last_clean_scrub_stamp":"2024-03-24T11:42:44.412955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:52.845399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":776,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404643+0000","last_change":"2024-03-24T11:42:59.392439+0000","last_active":"2024-03-24T11:42:59.404643+0000","last_peered":"2024-03-24T11:42:59.404643+0000","last_clean":"2024-03-24T11:42:59.404643+0000","last_became_active":"2024-03-24T11:41:49.149359+0000","last_became_peered":"2024-03-24T11:41:49.149359+0000","last_unstale":"2024-03-24T11:42:59.404643+0000","last_undegraded":"2024-03-24T11:42:59.404643+0000","last_fullsized":"2024-03-24T11:42:59.404643+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:59.392349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:39.212274+0000","last_clean_scrub_stamp":"2024-03-24T11:42:59.392349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:21.897033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1002,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012880+0000","last_change":"2024-03-24T11:42:45.765441+0000","last_active":"2024-03-24T11:43:00.012880+0000","last_peered":"2024-03-24T11:43:00.012880+0000","last_clean":"2024-03-24T11:43:00.012880+0000","last_became_active":"2024-03-24T11:41:12.168488+0000","last_became_peered":"2024-03-24T11:41:12.168488+0000","last_unstale":"2024-03-24T11:43:00.012880+0000","last_undegraded":"2024-03-24T11:43:00.012880+0000","last_fullsized":"2024-03-24T11:43:00.012880+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:45.765340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:32.567864+0000","last_clean_scrub_stamp":"2024-03-24T11:42:45.765340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.324863+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":1154,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976774+0000","last_change":"2024-03-24T11:42:57.287304+0000","last_active":"2024-03-24T11:42:59.976774+0000","last_peered":"2024-03-24T11:42:59.976774+0000","last_clean":"2024-03-24T11:42:59.976774+0000","last_became_active":"2024-03-24T11:41:12.181243+0000","last_became_peered":"2024-03-24T11:41:12.181243+0000","last_unstale":"2024-03-24T11:42:59.976774+0000","last_undegraded":"2024-03-24T11:42:59.976774+0000","last_fullsized":"2024-03-24T11:42:59.976774+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'60","last_scrub_stamp":"2024-03-24T11:42:57.287151+0000","last_deep_scrub":"421'60","last_deep_scrub_stamp":"2024-03-24T11:41:40.343360+0000","last_clean_scrub_stamp":"2024-03-24T11:42:57.287151+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:16.888131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":547,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013737+0000","last_change":"2024-03-24T11:42:47.008473+0000","last_active":"2024-03-24T11:43:00.013737+0000","last_peered":"2024-03-24T11:43:00.013737+0000","last_clean":"2024-03-24T11:43:00.013737+0000","last_became_active":"2024-03-24T11:42:23.701892+0000","last_became_peered":"2024-03-24T11:42:23.701892+0000","last_unstale":"2024-03-24T11:43:00.013737+0000","last_undegraded":"2024-03-24T11:43:00.013737+0000","last_fullsized":"2024-03-24T11:43:00.013737+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:47.008366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:30.990378+0000","last_clean_scrub_stamp":"2024-03-24T11:42:47.008366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:16.212319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1017,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404977+0000","last_change":"2024-03-24T11:42:28.372972+0000","last_active":"2024-03-24T11:42:59.404977+0000","last_peered":"2024-03-24T11:42:59.404977+0000","last_clean":"2024-03-24T11:42:59.404977+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:42:59.404977+0000","last_undegraded":"2024-03-24T11:42:59.404977+0000","last_fullsized":"2024-03-24T11:42:59.404977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:28.372893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:42:28.372893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:31.752046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":339,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013217+0000","last_change":"2024-03-24T11:42:23.705318+0000","last_active":"2024-03-24T11:43:00.013217+0000","last_peered":"2024-03-24T11:43:00.013217+0000","last_clean":"2024-03-24T11:43:00.013217+0000","last_became_active":"2024-03-24T11:42:23.704796+0000","last_became_peered":"2024-03-24T11:42:23.704796+0000","last_unstale":"2024-03-24T11:43:00.013217+0000","last_undegraded":"2024-03-24T11:43:00.013217+0000","last_fullsized":"2024-03-24T11:43:00.013217+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:55.379062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_clean_scrub_stamp":"2024-03-24T11:41:55.379062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:05.201594+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":784,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403571+0000","last_change":"2024-03-24T11:42:10.282085+0000","last_active":"2024-03-24T11:42:59.403571+0000","last_peered":"2024-03-24T11:42:59.403571+0000","last_clean":"2024-03-24T11:42:59.403571+0000","last_became_active":"2024-03-24T11:41:49.360640+0000","last_became_peered":"2024-03-24T11:41:49.360640+0000","last_unstale":"2024-03-24T11:42:59.403571+0000","last_undegraded":"2024-03-24T11:42:59.403571+0000","last_fullsized":"2024-03-24T11:42:59.403571+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:10.282008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.282008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:34.938772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"480'29","reported_seq":118,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576670+0000","last_change":"2024-03-24T11:42:56.057373+0000","last_active":"2024-03-24T11:42:59.576670+0000","last_peered":"2024-03-24T11:42:59.576670+0000","last_clean":"2024-03-24T11:42:59.576670+0000","last_became_active":"2024-03-24T11:41:20.441074+0000","last_became_peered":"2024-03-24T11:41:20.441074+0000","last_unstale":"2024-03-24T11:42:59.576670+0000","last_undegraded":"2024-03-24T11:42:59.576670+0000","last_fullsized":"2024-03-24T11:42:59.576670+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'29","last_scrub_stamp":"2024-03-24T11:42:56.057229+0000","last_deep_scrub":"480'29","last_deep_scrub_stamp":"2024-03-24T11:41:37.153709+0000","last_clean_scrub_stamp":"2024-03-24T11:42:56.057229+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:23.891977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":232,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013815+0000","last_change":"2024-03-24T11:42:26.847291+0000","last_active":"2024-03-24T11:43:00.013815+0000","last_peered":"2024-03-24T11:43:00.013815+0000","last_clean":"2024-03-24T11:43:00.013815+0000","last_became_active":"2024-03-24T11:42:23.695089+0000","last_became_peered":"2024-03-24T11:42:23.695089+0000","last_unstale":"2024-03-24T11:43:00.013815+0000","last_undegraded":"2024-03-24T11:43:00.013815+0000","last_fullsized":"2024-03-24T11:43:00.013815+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:26.847185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:42:26.847185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:48.668825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":968,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404391+0000","last_change":"2024-03-24T11:42:10.116945+0000","last_active":"2024-03-24T11:42:59.404391+0000","last_peered":"2024-03-24T11:42:59.404391+0000","last_clean":"2024-03-24T11:42:59.404391+0000","last_became_active":"2024-03-24T11:41:12.184410+0000","last_became_peered":"2024-03-24T11:41:12.184410+0000","last_unstale":"2024-03-24T11:42:59.404391+0000","last_undegraded":"2024-03-24T11:42:59.404391+0000","last_fullsized":"2024-03-24T11:42:59.404391+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:10.116836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.116836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:21.199601+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"503'15","reported_seq":725,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013532+0000","last_change":"2024-03-24T11:42:23.700475+0000","last_active":"2024-03-24T11:43:00.013532+0000","last_peered":"2024-03-24T11:43:00.013532+0000","last_clean":"2024-03-24T11:43:00.013532+0000","last_became_active":"2024-03-24T11:42:23.699377+0000","last_became_peered":"2024-03-24T11:42:23.699377+0000","last_unstale":"2024-03-24T11:43:00.013532+0000","last_undegraded":"2024-03-24T11:43:00.013532+0000","last_fullsized":"2024-03-24T11:43:00.013532+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'14","last_scrub_stamp":"2024-03-24T11:42:07.324847+0000","last_deep_scrub":"477'14","last_deep_scrub_stamp":"2024-03-24T11:42:07.324847+0000","last_clean_scrub_stamp":"2024-03-24T11:42:07.324847+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:36.364813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"421'46","reported_seq":1110,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401509+0000","last_change":"2024-03-24T11:42:28.353945+0000","last_active":"2024-03-24T11:42:59.401509+0000","last_peered":"2024-03-24T11:42:59.401509+0000","last_clean":"2024-03-24T11:42:59.401509+0000","last_became_active":"2024-03-24T11:41:12.171159+0000","last_became_peered":"2024-03-24T11:41:12.171159+0000","last_unstale":"2024-03-24T11:42:59.401509+0000","last_undegraded":"2024-03-24T11:42:59.401509+0000","last_fullsized":"2024-03-24T11:42:59.401509+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-24T11:42:28.353823+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:42:28.353823+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:47.808566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":436,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014022+0000","last_change":"2024-03-24T11:42:23.700259+0000","last_active":"2024-03-24T11:43:00.014022+0000","last_peered":"2024-03-24T11:43:00.014022+0000","last_clean":"2024-03-24T11:43:00.014022+0000","last_became_active":"2024-03-24T11:42:23.699896+0000","last_became_peered":"2024-03-24T11:42:23.699896+0000","last_unstale":"2024-03-24T11:43:00.014022+0000","last_undegraded":"2024-03-24T11:43:00.014022+0000","last_fullsized":"2024-03-24T11:43:00.014022+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:42:19.062303+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:42:19.062303+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:26.846716+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":990,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401969+0000","last_change":"2024-03-24T11:42:33.347314+0000","last_active":"2024-03-24T11:42:59.401969+0000","last_peered":"2024-03-24T11:42:59.401969+0000","last_clean":"2024-03-24T11:42:59.401969+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:42:59.401969+0000","last_undegraded":"2024-03-24T11:42:59.401969+0000","last_fullsized":"2024-03-24T11:42:59.401969+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:33.347218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:33.347218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:57.407696+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":1022,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404759+0000","last_change":"2024-03-24T11:42:15.893463+0000","last_active":"2024-03-24T11:42:59.404759+0000","last_peered":"2024-03-24T11:42:59.404759+0000","last_clean":"2024-03-24T11:42:59.404759+0000","last_became_active":"2024-03-24T11:41:12.183577+0000","last_became_peered":"2024-03-24T11:41:12.183577+0000","last_unstale":"2024-03-24T11:42:59.404759+0000","last_undegraded":"2024-03-24T11:42:59.404759+0000","last_fullsized":"2024-03-24T11:42:59.404759+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:15.893371+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:42:15.893371+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:22.725670+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1025,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976309+0000","last_change":"2024-03-24T11:42:48.340086+0000","last_active":"2024-03-24T11:42:59.976309+0000","last_peered":"2024-03-24T11:42:59.976309+0000","last_clean":"2024-03-24T11:42:59.976309+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:42:59.976309+0000","last_undegraded":"2024-03-24T11:42:59.976309+0000","last_fullsized":"2024-03-24T11:42:59.976309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:48.339988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:48.339988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:12.014567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":984,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402295+0000","last_change":"2024-03-24T11:42:02.172667+0000","last_active":"2024-03-24T11:42:59.402295+0000","last_peered":"2024-03-24T11:42:59.402295+0000","last_clean":"2024-03-24T11:42:59.402295+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:42:59.402295+0000","last_undegraded":"2024-03-24T11:42:59.402295+0000","last_fullsized":"2024-03-24T11:42:59.402295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:02.172558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:02.172558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:24.966201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":575,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013861+0000","last_change":"2024-03-24T11:42:23.702841+0000","last_active":"2024-03-24T11:43:00.013861+0000","last_peered":"2024-03-24T11:43:00.013861+0000","last_clean":"2024-03-24T11:43:00.013861+0000","last_became_active":"2024-03-24T11:42:23.702317+0000","last_became_peered":"2024-03-24T11:42:23.702317+0000","last_unstale":"2024-03-24T11:43:00.013861+0000","last_undegraded":"2024-03-24T11:43:00.013861+0000","last_fullsized":"2024-03-24T11:43:00.013861+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:57.246929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:24.442599+0000","last_clean_scrub_stamp":"2024-03-24T11:41:57.246929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:19.661822+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"421'25","reported_seq":1066,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014076+0000","last_change":"2024-03-24T11:42:18.451180+0000","last_active":"2024-03-24T11:43:00.014076+0000","last_peered":"2024-03-24T11:43:00.014076+0000","last_clean":"2024-03-24T11:43:00.014076+0000","last_became_active":"2024-03-24T11:41:12.175139+0000","last_became_peered":"2024-03-24T11:41:12.175139+0000","last_unstale":"2024-03-24T11:43:00.014076+0000","last_undegraded":"2024-03-24T11:43:00.014076+0000","last_fullsized":"2024-03-24T11:43:00.014076+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'25","last_scrub_stamp":"2024-03-24T11:42:18.451052+0000","last_deep_scrub":"421'25","last_deep_scrub_stamp":"2024-03-24T11:41:06.190172+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.451052+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.425395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":224,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013805+0000","last_change":"2024-03-24T11:42:23.700877+0000","last_active":"2024-03-24T11:43:00.013805+0000","last_peered":"2024-03-24T11:43:00.013805+0000","last_clean":"2024-03-24T11:43:00.013805+0000","last_became_active":"2024-03-24T11:42:23.700510+0000","last_became_peered":"2024-03-24T11:42:23.700510+0000","last_unstale":"2024-03-24T11:43:00.013805+0000","last_undegraded":"2024-03-24T11:43:00.013805+0000","last_fullsized":"2024-03-24T11:43:00.013805+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:55.105116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:55.105116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:06.529170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":971,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404705+0000","last_change":"2024-03-24T11:42:03.027873+0000","last_active":"2024-03-24T11:42:59.404705+0000","last_peered":"2024-03-24T11:42:59.404705+0000","last_clean":"2024-03-24T11:42:59.404705+0000","last_became_active":"2024-03-24T11:41:12.183024+0000","last_became_peered":"2024-03-24T11:41:12.183024+0000","last_unstale":"2024-03-24T11:42:59.404705+0000","last_undegraded":"2024-03-24T11:42:59.404705+0000","last_fullsized":"2024-03-24T11:42:59.404705+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:03.027789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:42:03.027789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:23.047893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1029,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404787+0000","last_change":"2024-03-24T11:42:26.319534+0000","last_active":"2024-03-24T11:42:59.404787+0000","last_peered":"2024-03-24T11:42:59.404787+0000","last_clean":"2024-03-24T11:42:59.404787+0000","last_became_active":"2024-03-24T11:41:20.421448+0000","last_became_peered":"2024-03-24T11:41:20.421448+0000","last_unstale":"2024-03-24T11:42:59.404787+0000","last_undegraded":"2024-03-24T11:42:59.404787+0000","last_fullsized":"2024-03-24T11:42:59.404787+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:26.319444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:26.319444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:30.966523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":1079,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402506+0000","last_change":"2024-03-24T11:42:39.439372+0000","last_active":"2024-03-24T11:42:59.402506+0000","last_peered":"2024-03-24T11:42:59.402506+0000","last_clean":"2024-03-24T11:42:59.402506+0000","last_became_active":"2024-03-24T11:41:12.174846+0000","last_became_peered":"2024-03-24T11:41:12.174846+0000","last_unstale":"2024-03-24T11:42:59.402506+0000","last_undegraded":"2024-03-24T11:42:59.402506+0000","last_fullsized":"2024-03-24T11:42:59.402506+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:39.439182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:39.439182+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:39.780431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1013,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013098+0000","last_change":"2024-03-24T11:42:38.579733+0000","last_active":"2024-03-24T11:43:00.013098+0000","last_peered":"2024-03-24T11:43:00.013098+0000","last_clean":"2024-03-24T11:43:00.013098+0000","last_became_active":"2024-03-24T11:41:12.169201+0000","last_became_peered":"2024-03-24T11:41:12.169201+0000","last_unstale":"2024-03-24T11:43:00.013098+0000","last_undegraded":"2024-03-24T11:43:00.013098+0000","last_fullsized":"2024-03-24T11:43:00.013098+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:38.579645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:38.579645+0000","last_clean_scrub_stamp":"2024-03-24T11:42:38.579645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.350314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"469'10","reported_seq":1287,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404459+0000","last_change":"2024-03-24T11:42:38.688123+0000","last_active":"2024-03-24T11:42:59.404459+0000","last_peered":"2024-03-24T11:42:59.404459+0000","last_clean":"2024-03-24T11:42:59.404459+0000","last_became_active":"2024-03-24T11:41:12.180989+0000","last_became_peered":"2024-03-24T11:41:12.180989+0000","last_unstale":"2024-03-24T11:42:59.404459+0000","last_undegraded":"2024-03-24T11:42:59.404459+0000","last_fullsized":"2024-03-24T11:42:59.404459+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'10","last_scrub_stamp":"2024-03-24T11:42:38.688025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:38.688025+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:51.896538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577531+0000","last_change":"2024-03-24T11:42:30.151531+0000","last_active":"2024-03-24T11:42:59.577531+0000","last_peered":"2024-03-24T11:42:59.577531+0000","last_clean":"2024-03-24T11:42:59.577531+0000","last_became_active":"2024-03-24T11:41:20.439380+0000","last_became_peered":"2024-03-24T11:41:20.439380+0000","last_unstale":"2024-03-24T11:42:59.577531+0000","last_undegraded":"2024-03-24T11:42:59.577531+0000","last_fullsized":"2024-03-24T11:42:59.577531+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:30.151432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:42:30.151432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.535371+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":1045,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404345+0000","last_change":"2024-03-24T11:41:57.084015+0000","last_active":"2024-03-24T11:42:59.404345+0000","last_peered":"2024-03-24T11:42:59.404345+0000","last_clean":"2024-03-24T11:42:59.404345+0000","last_became_active":"2024-03-24T11:41:12.181695+0000","last_became_peered":"2024-03-24T11:41:12.181695+0000","last_unstale":"2024-03-24T11:42:59.404345+0000","last_undegraded":"2024-03-24T11:42:59.404345+0000","last_fullsized":"2024-03-24T11:42:59.404345+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:41:57.083911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:57.083911+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:10.767591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":232,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013691+0000","last_change":"2024-03-24T11:42:43.033863+0000","last_active":"2024-03-24T11:43:00.013691+0000","last_peered":"2024-03-24T11:43:00.013691+0000","last_clean":"2024-03-24T11:43:00.013691+0000","last_became_active":"2024-03-24T11:42:23.703737+0000","last_became_peered":"2024-03-24T11:42:23.703737+0000","last_unstale":"2024-03-24T11:43:00.013691+0000","last_undegraded":"2024-03-24T11:43:00.013691+0000","last_fullsized":"2024-03-24T11:43:00.013691+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:43.033756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:43.033756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:00.813656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"504'9","reported_seq":1291,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402328+0000","last_change":"2024-03-24T11:42:27.314508+0000","last_active":"2024-03-24T11:42:59.402328+0000","last_peered":"2024-03-24T11:42:59.402328+0000","last_clean":"2024-03-24T11:42:59.402328+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:42:59.402328+0000","last_undegraded":"2024-03-24T11:42:59.402328+0000","last_fullsized":"2024-03-24T11:42:59.402328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'9","last_scrub_stamp":"2024-03-24T11:42:27.314365+0000","last_deep_scrub":"504'9","last_deep_scrub_stamp":"2024-03-24T11:42:27.314365+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.314365+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:51.825564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":992,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013902+0000","last_change":"2024-03-24T11:42:37.676031+0000","last_active":"2024-03-24T11:43:00.013902+0000","last_peered":"2024-03-24T11:43:00.013902+0000","last_clean":"2024-03-24T11:43:00.013902+0000","last_became_active":"2024-03-24T11:41:12.175157+0000","last_became_peered":"2024-03-24T11:41:12.175157+0000","last_unstale":"2024-03-24T11:43:00.013902+0000","last_undegraded":"2024-03-24T11:43:00.013902+0000","last_fullsized":"2024-03-24T11:43:00.013902+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:37.675933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:37.675933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:47.284048+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'32","reported_seq":113,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576991+0000","last_change":"2024-03-24T11:42:54.028771+0000","last_active":"2024-03-24T11:42:59.576991+0000","last_peered":"2024-03-24T11:42:59.576991+0000","last_clean":"2024-03-24T11:42:59.576991+0000","last_became_active":"2024-03-24T11:41:20.430421+0000","last_became_peered":"2024-03-24T11:41:20.430421+0000","last_unstale":"2024-03-24T11:42:59.576991+0000","last_undegraded":"2024-03-24T11:42:59.576991+0000","last_fullsized":"2024-03-24T11:42:59.576991+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:42:54.028629+0000","last_deep_scrub":"421'32","last_deep_scrub_stamp":"2024-03-24T11:42:54.028629+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.028629+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:23.508120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576940+0000","last_change":"2024-03-24T11:42:35.894966+0000","last_active":"2024-03-24T11:42:59.576940+0000","last_peered":"2024-03-24T11:42:59.576940+0000","last_clean":"2024-03-24T11:42:59.576940+0000","last_became_active":"2024-03-24T11:41:20.434013+0000","last_became_peered":"2024-03-24T11:41:20.434013+0000","last_unstale":"2024-03-24T11:42:59.576940+0000","last_undegraded":"2024-03-24T11:42:59.576940+0000","last_fullsized":"2024-03-24T11:42:59.576940+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:35.894863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:35.894863+0000","last_clean_scrub_stamp":"2024-03-24T11:42:35.894863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:49.904901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1001,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013617+0000","last_change":"2024-03-24T11:42:27.088898+0000","last_active":"2024-03-24T11:43:00.013617+0000","last_peered":"2024-03-24T11:43:00.013617+0000","last_clean":"2024-03-24T11:43:00.013617+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:43:00.013617+0000","last_undegraded":"2024-03-24T11:43:00.013617+0000","last_fullsized":"2024-03-24T11:43:00.013617+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:27.088809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.088809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.637250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":964,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403627+0000","last_change":"2024-03-24T11:42:58.554146+0000","last_active":"2024-03-24T11:42:59.403627+0000","last_peered":"2024-03-24T11:42:59.403627+0000","last_clean":"2024-03-24T11:42:59.403627+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:42:59.403627+0000","last_undegraded":"2024-03-24T11:42:59.403627+0000","last_fullsized":"2024-03-24T11:42:59.403627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:58.554018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:58.554018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:21.230624+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":1031,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013803+0000","last_change":"2024-03-24T11:42:21.147328+0000","last_active":"2024-03-24T11:43:00.013803+0000","last_peered":"2024-03-24T11:43:00.013803+0000","last_clean":"2024-03-24T11:43:00.013803+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:43:00.013803+0000","last_undegraded":"2024-03-24T11:43:00.013803+0000","last_fullsized":"2024-03-24T11:43:00.013803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:21.147218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:21.147218+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:28.683459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":550,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013637+0000","last_change":"2024-03-24T11:42:23.701309+0000","last_active":"2024-03-24T11:43:00.013637+0000","last_peered":"2024-03-24T11:43:00.013637+0000","last_clean":"2024-03-24T11:43:00.013637+0000","last_became_active":"2024-03-24T11:42:23.700882+0000","last_became_peered":"2024-03-24T11:42:23.700882+0000","last_unstale":"2024-03-24T11:43:00.013637+0000","last_undegraded":"2024-03-24T11:43:00.013637+0000","last_fullsized":"2024-03-24T11:43:00.013637+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:48.115600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:48.115600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":960,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404121+0000","last_change":"2024-03-24T11:41:56.046644+0000","last_active":"2024-03-24T11:42:59.404121+0000","last_peered":"2024-03-24T11:42:59.404121+0000","last_clean":"2024-03-24T11:42:59.404121+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:42:59.404121+0000","last_undegraded":"2024-03-24T11:42:59.404121+0000","last_fullsized":"2024-03-24T11:42:59.404121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:56.046564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:41:56.046564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":989,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:03.613976+0000","last_change":"2024-03-24T11:43:03.614001+0000","last_active":"2024-03-24T11:43:03.613976+0000","last_peered":"2024-03-24T11:43:03.613976+0000","last_clean":"2024-03-24T11:43:03.613976+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:43:03.613976+0000","last_undegraded":"2024-03-24T11:43:03.613976+0000","last_fullsized":"2024-03-24T11:43:03.613976+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:03.613874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:03.613874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:26.395898+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":1043,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405019+0000","last_change":"2024-03-24T11:41:58.093979+0000","last_active":"2024-03-24T11:42:59.405019+0000","last_peered":"2024-03-24T11:42:59.405019+0000","last_clean":"2024-03-24T11:42:59.405019+0000","last_became_active":"2024-03-24T11:41:12.181398+0000","last_became_peered":"2024-03-24T11:41:12.181398+0000","last_unstale":"2024-03-24T11:42:59.405019+0000","last_undegraded":"2024-03-24T11:42:59.405019+0000","last_fullsized":"2024-03-24T11:42:59.405019+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:41:58.093860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:58.093860+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:23.139381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":921,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404560+0000","last_change":"2024-03-24T11:42:48.418666+0000","last_active":"2024-03-24T11:42:59.404560+0000","last_peered":"2024-03-24T11:42:59.404560+0000","last_clean":"2024-03-24T11:42:59.404560+0000","last_became_active":"2024-03-24T11:41:48.525793+0000","last_became_peered":"2024-03-24T11:41:48.525793+0000","last_unstale":"2024-03-24T11:42:59.404560+0000","last_undegraded":"2024-03-24T11:42:59.404560+0000","last_fullsized":"2024-03-24T11:42:59.404560+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:48.418020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:48.418020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:07.313849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":232,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013338+0000","last_change":"2024-03-24T11:42:36.967791+0000","last_active":"2024-03-24T11:43:00.013338+0000","last_peered":"2024-03-24T11:43:00.013338+0000","last_clean":"2024-03-24T11:43:00.013338+0000","last_became_active":"2024-03-24T11:42:23.697581+0000","last_became_peered":"2024-03-24T11:42:23.697581+0000","last_unstale":"2024-03-24T11:43:00.013338+0000","last_undegraded":"2024-03-24T11:43:00.013338+0000","last_fullsized":"2024-03-24T11:43:00.013338+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:36.967679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.967679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:04.305581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":964,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404298+0000","last_change":"2024-03-24T11:42:14.989707+0000","last_active":"2024-03-24T11:42:59.404298+0000","last_peered":"2024-03-24T11:42:59.404298+0000","last_clean":"2024-03-24T11:42:59.404298+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:42:59.404298+0000","last_undegraded":"2024-03-24T11:42:59.404298+0000","last_fullsized":"2024-03-24T11:42:59.404298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:14.989611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:14.989611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:31.105756+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"421'14","reported_seq":984,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403300+0000","last_change":"2024-03-24T11:41:52.126576+0000","last_active":"2024-03-24T11:42:59.403300+0000","last_peered":"2024-03-24T11:42:59.403300+0000","last_clean":"2024-03-24T11:42:59.403300+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:42:59.403300+0000","last_undegraded":"2024-03-24T11:42:59.403300+0000","last_fullsized":"2024-03-24T11:42:59.403300+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'14","last_scrub_stamp":"2024-03-24T11:41:52.126490+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:41:52.126490+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:04.530316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1000,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013538+0000","last_change":"2024-03-24T11:42:35.601012+0000","last_active":"2024-03-24T11:43:00.013538+0000","last_peered":"2024-03-24T11:43:00.013538+0000","last_clean":"2024-03-24T11:43:00.013538+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:43:00.013538+0000","last_undegraded":"2024-03-24T11:43:00.013538+0000","last_fullsized":"2024-03-24T11:43:00.013538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:35.600871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:35.600871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.131629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1033,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976607+0000","last_change":"2024-03-24T11:42:04.285879+0000","last_active":"2024-03-24T11:42:59.976607+0000","last_peered":"2024-03-24T11:42:59.976607+0000","last_clean":"2024-03-24T11:42:59.976607+0000","last_became_active":"2024-03-24T11:41:20.433021+0000","last_became_peered":"2024-03-24T11:41:20.433021+0000","last_unstale":"2024-03-24T11:42:59.976607+0000","last_undegraded":"2024-03-24T11:42:59.976607+0000","last_fullsized":"2024-03-24T11:42:59.976607+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:04.285779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_clean_scrub_stamp":"2024-03-24T11:42:04.285779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:18.062036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":347,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.867222+0000","last_change":"2024-03-24T11:43:00.867247+0000","last_active":"2024-03-24T11:43:00.867222+0000","last_peered":"2024-03-24T11:43:00.867222+0000","last_clean":"2024-03-24T11:43:00.867222+0000","last_became_active":"2024-03-24T11:42:23.701966+0000","last_became_peered":"2024-03-24T11:42:23.701966+0000","last_unstale":"2024-03-24T11:43:00.867222+0000","last_undegraded":"2024-03-24T11:43:00.867222+0000","last_fullsized":"2024-03-24T11:43:00.867222+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:00.867112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:00.867112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:03.418480+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"421'32","reported_seq":1062,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977214+0000","last_change":"2024-03-24T11:42:23.378682+0000","last_active":"2024-03-24T11:42:59.977214+0000","last_peered":"2024-03-24T11:42:59.977214+0000","last_clean":"2024-03-24T11:42:59.977214+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:42:59.977214+0000","last_undegraded":"2024-03-24T11:42:59.977214+0000","last_fullsized":"2024-03-24T11:42:59.977214+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:42:23.378553+0000","last_deep_scrub":"421'32","last_deep_scrub_stamp":"2024-03-24T11:42:23.378553+0000","last_clean_scrub_stamp":"2024-03-24T11:42:23.378553+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:30.934465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":967,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403294+0000","last_change":"2024-03-24T11:41:51.145690+0000","last_active":"2024-03-24T11:42:59.403294+0000","last_peered":"2024-03-24T11:42:59.403294+0000","last_clean":"2024-03-24T11:42:59.403294+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:42:59.403294+0000","last_undegraded":"2024-03-24T11:42:59.403294+0000","last_fullsized":"2024-03-24T11:42:59.403294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:51.145592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:51.145592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:05.118936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1007,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014170+0000","last_change":"2024-03-24T11:42:50.675604+0000","last_active":"2024-03-24T11:43:00.014170+0000","last_peered":"2024-03-24T11:43:00.014170+0000","last_clean":"2024-03-24T11:43:00.014170+0000","last_became_active":"2024-03-24T11:41:12.174243+0000","last_became_peered":"2024-03-24T11:41:12.174243+0000","last_unstale":"2024-03-24T11:43:00.014170+0000","last_undegraded":"2024-03-24T11:43:00.014170+0000","last_fullsized":"2024-03-24T11:43:00.014170+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:50.675523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:42:50.675523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:09.952710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1008,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:02.517662+0000","last_change":"2024-03-24T11:43:02.517686+0000","last_active":"2024-03-24T11:43:02.517662+0000","last_peered":"2024-03-24T11:43:02.517662+0000","last_clean":"2024-03-24T11:43:02.517662+0000","last_became_active":"2024-03-24T11:41:20.434440+0000","last_became_peered":"2024-03-24T11:41:20.434440+0000","last_unstale":"2024-03-24T11:43:02.517662+0000","last_undegraded":"2024-03-24T11:43:02.517662+0000","last_fullsized":"2024-03-24T11:43:02.517662+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:02.517584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:40.280347+0000","last_clean_scrub_stamp":"2024-03-24T11:43:02.517584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:30.427027+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'30","reported_seq":1106,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014022+0000","last_change":"2024-03-24T11:42:54.726626+0000","last_active":"2024-03-24T11:43:00.014022+0000","last_peered":"2024-03-24T11:43:00.014022+0000","last_clean":"2024-03-24T11:43:00.014022+0000","last_became_active":"2024-03-24T11:41:12.176633+0000","last_became_peered":"2024-03-24T11:41:12.176633+0000","last_unstale":"2024-03-24T11:43:00.014022+0000","last_undegraded":"2024-03-24T11:43:00.014022+0000","last_fullsized":"2024-03-24T11:43:00.014022+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:42:54.726527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.726527+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.033482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":1029,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012753+0000","last_change":"2024-03-24T11:42:04.123992+0000","last_active":"2024-03-24T11:43:00.012753+0000","last_peered":"2024-03-24T11:43:00.012753+0000","last_clean":"2024-03-24T11:43:00.012753+0000","last_became_active":"2024-03-24T11:41:20.436875+0000","last_became_peered":"2024-03-24T11:41:20.436875+0000","last_unstale":"2024-03-24T11:43:00.012753+0000","last_undegraded":"2024-03-24T11:43:00.012753+0000","last_fullsized":"2024-03-24T11:43:00.012753+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:42:04.123899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:04.123899+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:19.688388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"477'13","reported_seq":1355,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977116+0000","last_change":"2024-03-24T11:42:19.445563+0000","last_active":"2024-03-24T11:42:59.977116+0000","last_peered":"2024-03-24T11:42:59.977116+0000","last_clean":"2024-03-24T11:42:59.977116+0000","last_became_active":"2024-03-24T11:41:20.431908+0000","last_became_peered":"2024-03-24T11:41:20.431908+0000","last_unstale":"2024-03-24T11:42:59.977116+0000","last_undegraded":"2024-03-24T11:42:59.977116+0000","last_fullsized":"2024-03-24T11:42:59.977116+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'13","last_scrub_stamp":"2024-03-24T11:42:19.445321+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:42:19.445321+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:49.130407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":584,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013631+0000","last_change":"2024-03-24T11:42:23.704163+0000","last_active":"2024-03-24T11:43:00.013631+0000","last_peered":"2024-03-24T11:43:00.013631+0000","last_clean":"2024-03-24T11:43:00.013631+0000","last_became_active":"2024-03-24T11:42:23.703678+0000","last_became_peered":"2024-03-24T11:42:23.703678+0000","last_unstale":"2024-03-24T11:43:00.013631+0000","last_undegraded":"2024-03-24T11:43:00.013631+0000","last_fullsized":"2024-03-24T11:43:00.013631+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:41:53.139987+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:41:53.139987+0000","last_clean_scrub_stamp":"2024-03-24T11:41:53.139987+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:09.895781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"421'24","reported_seq":1032,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403816+0000","last_change":"2024-03-24T11:41:32.341653+0000","last_active":"2024-03-24T11:42:59.403816+0000","last_peered":"2024-03-24T11:42:59.403816+0000","last_clean":"2024-03-24T11:42:59.403816+0000","last_became_active":"2024-03-24T11:41:20.436384+0000","last_became_peered":"2024-03-24T11:41:20.436384+0000","last_unstale":"2024-03-24T11:42:59.403816+0000","last_undegraded":"2024-03-24T11:42:59.403816+0000","last_fullsized":"2024-03-24T11:42:59.403816+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'24","last_scrub_stamp":"2024-03-24T11:41:32.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:32.341564+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576715+0000","last_change":"2024-03-24T11:42:27.987032+0000","last_active":"2024-03-24T11:42:59.576715+0000","last_peered":"2024-03-24T11:42:59.576715+0000","last_clean":"2024-03-24T11:42:59.576715+0000","last_became_active":"2024-03-24T11:41:20.427341+0000","last_became_peered":"2024-03-24T11:41:20.427341+0000","last_unstale":"2024-03-24T11:42:59.576715+0000","last_undegraded":"2024-03-24T11:42:59.576715+0000","last_fullsized":"2024-03-24T11:42:59.576715+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:27.986921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:11.852848+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.986921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:43.072514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576943+0000","last_change":"2024-03-24T11:42:55.054792+0000","last_active":"2024-03-24T11:42:59.576943+0000","last_peered":"2024-03-24T11:42:59.576943+0000","last_clean":"2024-03-24T11:42:59.576943+0000","last_became_active":"2024-03-24T11:41:20.439681+0000","last_became_peered":"2024-03-24T11:41:20.439681+0000","last_unstale":"2024-03-24T11:42:59.576943+0000","last_undegraded":"2024-03-24T11:42:59.576943+0000","last_fullsized":"2024-03-24T11:42:59.576943+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:55.054677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:55.054677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:15.892918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":960,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404470+0000","last_change":"2024-03-24T11:42:12.097323+0000","last_active":"2024-03-24T11:42:59.404470+0000","last_peered":"2024-03-24T11:42:59.404470+0000","last_clean":"2024-03-24T11:42:59.404470+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:42:59.404470+0000","last_undegraded":"2024-03-24T11:42:59.404470+0000","last_fullsized":"2024-03-24T11:42:59.404470+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:12.097225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:12.097225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:28.933272+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"421'39","reported_seq":1081,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404477+0000","last_change":"2024-03-24T11:42:34.388539+0000","last_active":"2024-03-24T11:42:59.404477+0000","last_peered":"2024-03-24T11:42:59.404477+0000","last_clean":"2024-03-24T11:42:59.404477+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:42:59.404477+0000","last_undegraded":"2024-03-24T11:42:59.404477+0000","last_fullsized":"2024-03-24T11:42:59.404477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:42:34.388433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:34.388433+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.868559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1024,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013107+0000","last_change":"2024-03-24T11:42:47.634770+0000","last_active":"2024-03-24T11:43:00.013107+0000","last_peered":"2024-03-24T11:43:00.013107+0000","last_clean":"2024-03-24T11:43:00.013107+0000","last_became_active":"2024-03-24T11:41:20.431934+0000","last_became_peered":"2024-03-24T11:41:20.431934+0000","last_unstale":"2024-03-24T11:43:00.013107+0000","last_undegraded":"2024-03-24T11:43:00.013107+0000","last_fullsized":"2024-03-24T11:43:00.013107+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:47.634682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:47.634682+0000","last_clean_scrub_stamp":"2024-03-24T11:42:47.634682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:55.277468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1004,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014069+0000","last_change":"2024-03-24T11:42:49.144640+0000","last_active":"2024-03-24T11:43:00.014069+0000","last_peered":"2024-03-24T11:43:00.014069+0000","last_clean":"2024-03-24T11:43:00.014069+0000","last_became_active":"2024-03-24T11:41:20.439167+0000","last_became_peered":"2024-03-24T11:41:20.439167+0000","last_unstale":"2024-03-24T11:43:00.014069+0000","last_undegraded":"2024-03-24T11:43:00.014069+0000","last_fullsized":"2024-03-24T11:43:00.014069+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:49.144471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:42:49.144471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:54.634464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577238+0000","last_change":"2024-03-24T11:41:59.110867+0000","last_active":"2024-03-24T11:42:59.577238+0000","last_peered":"2024-03-24T11:42:59.577238+0000","last_clean":"2024-03-24T11:42:59.577238+0000","last_became_active":"2024-03-24T11:41:20.436845+0000","last_became_peered":"2024-03-24T11:41:20.436845+0000","last_unstale":"2024-03-24T11:42:59.577238+0000","last_undegraded":"2024-03-24T11:42:59.577238+0000","last_fullsized":"2024-03-24T11:42:59.577238+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:59.110715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:41:59.110715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:23.548564+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"421'9","reported_seq":99,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577178+0000","last_change":"2024-03-24T11:42:20.031601+0000","last_active":"2024-03-24T11:42:59.577178+0000","last_peered":"2024-03-24T11:42:59.577178+0000","last_clean":"2024-03-24T11:42:59.577178+0000","last_became_active":"2024-03-24T11:41:20.439702+0000","last_became_peered":"2024-03-24T11:41:20.439702+0000","last_unstale":"2024-03-24T11:42:59.577178+0000","last_undegraded":"2024-03-24T11:42:59.577178+0000","last_fullsized":"2024-03-24T11:42:59.577178+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9","last_scrub_stamp":"2024-03-24T11:42:20.031489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:20.031489+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:41.536673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1004,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402175+0000","last_change":"2024-03-24T11:41:59.087942+0000","last_active":"2024-03-24T11:42:59.402175+0000","last_peered":"2024-03-24T11:42:59.402175+0000","last_clean":"2024-03-24T11:42:59.402175+0000","last_became_active":"2024-03-24T11:41:20.423037+0000","last_became_peered":"2024-03-24T11:41:20.423037+0000","last_unstale":"2024-03-24T11:42:59.402175+0000","last_undegraded":"2024-03-24T11:42:59.402175+0000","last_fullsized":"2024-03-24T11:42:59.402175+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:59.087816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:41:59.087816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:24.054613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":224,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013465+0000","last_change":"2024-03-24T11:42:23.693015+0000","last_active":"2024-03-24T11:43:00.013465+0000","last_peered":"2024-03-24T11:43:00.013465+0000","last_clean":"2024-03-24T11:43:00.013465+0000","last_became_active":"2024-03-24T11:42:23.692534+0000","last_became_peered":"2024-03-24T11:42:23.692534+0000","last_unstale":"2024-03-24T11:43:00.013465+0000","last_undegraded":"2024-03-24T11:43:00.013465+0000","last_fullsized":"2024-03-24T11:43:00.013465+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:52.162832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:41:52.162832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:01.616558+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":980,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401756+0000","last_change":"2024-03-24T11:42:19.322923+0000","last_active":"2024-03-24T11:42:59.401756+0000","last_peered":"2024-03-24T11:42:59.401756+0000","last_clean":"2024-03-24T11:42:59.401756+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:42:59.401756+0000","last_undegraded":"2024-03-24T11:42:59.401756+0000","last_fullsized":"2024-03-24T11:42:59.401756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:19.322803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:42:19.322803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:26.121082+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"421'21","reported_seq":103,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576849+0000","last_change":"2024-03-24T11:41:46.167562+0000","last_active":"2024-03-24T11:42:59.576849+0000","last_peered":"2024-03-24T11:42:59.576849+0000","last_clean":"2024-03-24T11:42:59.576849+0000","last_became_active":"2024-03-24T11:41:20.442581+0000","last_became_peered":"2024-03-24T11:41:20.442581+0000","last_unstale":"2024-03-24T11:42:59.576849+0000","last_undegraded":"2024-03-24T11:42:59.576849+0000","last_fullsized":"2024-03-24T11:42:59.576849+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:41:46.167454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:46.167454+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.114702+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1012,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402237+0000","last_change":"2024-03-24T11:42:46.322506+0000","last_active":"2024-03-24T11:42:59.402237+0000","last_peered":"2024-03-24T11:42:59.402237+0000","last_clean":"2024-03-24T11:42:59.402237+0000","last_became_active":"2024-03-24T11:41:20.432122+0000","last_became_peered":"2024-03-24T11:41:20.432122+0000","last_unstale":"2024-03-24T11:42:59.402237+0000","last_undegraded":"2024-03-24T11:42:59.402237+0000","last_fullsized":"2024-03-24T11:42:59.402237+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:46.322399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:42:46.322399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.598188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1010,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401870+0000","last_change":"2024-03-24T11:42:04.185684+0000","last_active":"2024-03-24T11:42:59.401870+0000","last_peered":"2024-03-24T11:42:59.401870+0000","last_clean":"2024-03-24T11:42:59.401870+0000","last_became_active":"2024-03-24T11:41:12.168113+0000","last_became_peered":"2024-03-24T11:41:12.168113+0000","last_unstale":"2024-03-24T11:42:59.401870+0000","last_undegraded":"2024-03-24T11:42:59.401870+0000","last_fullsized":"2024-03-24T11:42:59.401870+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:04.185585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:42:04.185585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:22.138511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":992,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013031+0000","last_change":"2024-03-24T11:41:46.401947+0000","last_active":"2024-03-24T11:43:00.013031+0000","last_peered":"2024-03-24T11:43:00.013031+0000","last_clean":"2024-03-24T11:43:00.013031+0000","last_became_active":"2024-03-24T11:41:12.172311+0000","last_became_peered":"2024-03-24T11:41:12.172311+0000","last_unstale":"2024-03-24T11:43:00.013031+0000","last_undegraded":"2024-03-24T11:43:00.013031+0000","last_fullsized":"2024-03-24T11:43:00.013031+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:46.401832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:41:46.401832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:09.185024+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'39","reported_seq":1024,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405024+0000","last_change":"2024-03-24T11:42:44.623567+0000","last_active":"2024-03-24T11:42:59.405024+0000","last_peered":"2024-03-24T11:42:59.405024+0000","last_clean":"2024-03-24T11:42:59.405024+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:42:59.405024+0000","last_undegraded":"2024-03-24T11:42:59.405024+0000","last_fullsized":"2024-03-24T11:42:59.405024+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:42:44.623454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:44.623454+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:55.873880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":970,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403597+0000","last_change":"2024-03-24T11:42:36.656857+0000","last_active":"2024-03-24T11:42:59.403597+0000","last_peered":"2024-03-24T11:42:59.403597+0000","last_clean":"2024-03-24T11:42:59.403597+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:42:59.403597+0000","last_undegraded":"2024-03-24T11:42:59.403597+0000","last_fullsized":"2024-03-24T11:42:59.403597+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:36.656695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.656695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:01.226380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576442+0000","last_change":"2024-03-24T11:42:39.885660+0000","last_active":"2024-03-24T11:42:59.576442+0000","last_peered":"2024-03-24T11:42:59.576442+0000","last_clean":"2024-03-24T11:42:59.576442+0000","last_became_active":"2024-03-24T11:41:20.429824+0000","last_became_peered":"2024-03-24T11:41:20.429824+0000","last_unstale":"2024-03-24T11:42:59.576442+0000","last_undegraded":"2024-03-24T11:42:59.576442+0000","last_fullsized":"2024-03-24T11:42:59.576442+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:39.885558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:39.885558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:07.460919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":967,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404179+0000","last_change":"2024-03-24T11:42:23.043823+0000","last_active":"2024-03-24T11:42:59.404179+0000","last_peered":"2024-03-24T11:42:59.404179+0000","last_clean":"2024-03-24T11:42:59.404179+0000","last_became_active":"2024-03-24T11:41:12.185025+0000","last_became_peered":"2024-03-24T11:41:12.185025+0000","last_unstale":"2024-03-24T11:42:59.404179+0000","last_undegraded":"2024-03-24T11:42:59.404179+0000","last_fullsized":"2024-03-24T11:42:59.404179+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:23.043726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:23.043726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:34.641398+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":103,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576442+0000","last_change":"2024-03-24T11:41:49.143332+0000","last_active":"2024-03-24T11:42:59.576442+0000","last_peered":"2024-03-24T11:42:59.576442+0000","last_clean":"2024-03-24T11:42:59.576442+0000","last_became_active":"2024-03-24T11:41:20.431239+0000","last_became_peered":"2024-03-24T11:41:20.431239+0000","last_unstale":"2024-03-24T11:42:59.576442+0000","last_undegraded":"2024-03-24T11:42:59.576442+0000","last_fullsized":"2024-03-24T11:42:59.576442+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:41:49.143223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:49.143223+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:01.339933+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":994,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013733+0000","last_change":"2024-03-24T11:41:50.398182+0000","last_active":"2024-03-24T11:43:00.013733+0000","last_peered":"2024-03-24T11:43:00.013733+0000","last_clean":"2024-03-24T11:43:00.013733+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:43:00.013733+0000","last_undegraded":"2024-03-24T11:43:00.013733+0000","last_fullsized":"2024-03-24T11:43:00.013733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:50.398096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:50.398096+0000","last_clean_scrub_stamp":"2024-03-24T11:41:50.398096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:16.468009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1004,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012984+0000","last_change":"2024-03-24T11:42:20.180676+0000","last_active":"2024-03-24T11:43:00.012984+0000","last_peered":"2024-03-24T11:43:00.012984+0000","last_clean":"2024-03-24T11:43:00.012984+0000","last_became_active":"2024-03-24T11:41:12.181763+0000","last_became_peered":"2024-03-24T11:41:12.181763+0000","last_unstale":"2024-03-24T11:43:00.012984+0000","last_undegraded":"2024-03-24T11:43:00.012984+0000","last_fullsized":"2024-03-24T11:43:00.012984+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:20.180604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:20.180604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.552068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:01.057616+0000","last_change":"2024-03-24T11:43:01.057640+0000","last_active":"2024-03-24T11:43:01.057616+0000","last_peered":"2024-03-24T11:43:01.057616+0000","last_clean":"2024-03-24T11:43:01.057616+0000","last_became_active":"2024-03-24T11:41:20.437643+0000","last_became_peered":"2024-03-24T11:41:20.437643+0000","last_unstale":"2024-03-24T11:43:01.057616+0000","last_undegraded":"2024-03-24T11:43:01.057616+0000","last_fullsized":"2024-03-24T11:43:01.057616+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:01.057531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:01.057531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:05.222308+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":670,"num_read_kb":665,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":6694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1046,"ondisk_log_size":1046,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070759,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799000,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946152,"statfs":{"total":95995035648,"available":95176859648,"internally_reserved":0,"allocated":1519616,"data_stored":1040959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":816642507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365100,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713996,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93031156,"statfs":{"total":95995035648,"available":95263903744,"internally_reserved":0,"allocated":1179648,"data_stored":713500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938928,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806224,"statfs":{"total":95995035648,"available":95033573376,"internally_reserved":0,"allocated":1220608,"data_stored":714337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738699,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840356,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92904796,"statfs":{"total":95995035648,"available":95134511104,"internally_reserved":0,"allocated":1204224,"data_stored":713144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411649,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722588,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93022564,"statfs":{"total":95995035648,"available":95255105536,"internally_reserved":0,"allocated":1454080,"data_stored":975483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150010,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881180,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92863972,"statfs":{"total":95995035648,"available":95092707328,"internally_reserved":0,"allocated":1789952,"data_stored":1303429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":900528583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986486,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569632,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175520,"statfs":{"total":95995035648,"available":95411732480,"internally_reserved":0,"allocated":1204224,"data_stored":713707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659441,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849248,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92895904,"statfs":{"total":95995035648,"available":95125405696,"internally_reserved":0,"allocated":1794048,"data_stored":1303787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:43:09.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:43:09.382 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 4.18 scrub starts 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 4.18 scrub ok 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 6.d scrub starts 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: from='client.26177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 5.13 scrub starts 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 5.13 scrub ok 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 6.d scrub ok 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: pgmap v1140: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:09.835 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 6.f scrub starts 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[29608]: 6.f scrub ok 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 4.18 scrub starts 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 4.18 scrub ok 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 6.d scrub starts 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: from='client.26177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 5.13 scrub starts 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 5.13 scrub ok 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 6.d scrub ok 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: pgmap v1140: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 6.f scrub starts 2024-03-24T11:43:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:09 smithi179 ceph-mon[37907]: 6.f scrub ok 2024-03-24T11:43:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 4.18 scrub starts 2024-03-24T11:43:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 4.18 scrub ok 2024-03-24T11:43:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 6.d scrub starts 2024-03-24T11:43:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: from='client.26177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:43:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 5.13 scrub starts 2024-03-24T11:43:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 5.13 scrub ok 2024-03-24T11:43:09.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 6.d scrub ok 2024-03-24T11:43:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: pgmap v1140: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 6.f scrub starts 2024-03-24T11:43:09.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:09 smithi181 ceph-mon[33998]: 6.f scrub ok 2024-03-24T11:43:10.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:10 smithi179 ceph-mon[29608]: purged_snaps scrub starts 2024-03-24T11:43:10.868 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:10 smithi179 ceph-mon[29608]: purged_snaps scrub ok 2024-03-24T11:43:10.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:10 smithi179 ceph-mon[37907]: purged_snaps scrub starts 2024-03-24T11:43:10.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:10 smithi179 ceph-mon[37907]: purged_snaps scrub ok 2024-03-24T11:43:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:10 smithi181 ceph-mon[33998]: purged_snaps scrub starts 2024-03-24T11:43:10.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:10 smithi181 ceph-mon[33998]: purged_snaps scrub ok 2024-03-24T11:43:11.317 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-24T11:43:11.319 INFO:teuthology.orchestra.run.smithi179.stderr:dumped all 2024-03-24T11:43:11.786 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[29608]: 6.18 scrub starts 2024-03-24T11:43:11.786 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[29608]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-24T11:43:11.786 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[29608]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T11:43:11.786 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[29608]: 6.18 scrub ok 2024-03-24T11:43:11.787 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[37907]: 6.18 scrub starts 2024-03-24T11:43:11.787 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[37907]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-24T11:43:11.787 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[37907]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T11:43:11.787 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:11 smithi179 ceph-mon[37907]: 6.18 scrub ok 2024-03-24T11:43:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:11 smithi181 ceph-mon[33998]: 6.18 scrub starts 2024-03-24T11:43:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:11 smithi181 ceph-mon[33998]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-24T11:43:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:11 smithi181 ceph-mon[33998]: osdmap e513: 8 total, 8 up, 8 in 2024-03-24T11:43:11.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:11 smithi181 ceph-mon[33998]: 6.18 scrub ok 2024-03-24T11:43:11.984 INFO:teuthology.orchestra.run.smithi179.stdout:{"pg_ready":true,"pg_map":{"version":1142,"stamp":"2024-03-24T11:43:10.545402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2238,"num_read_kb":1990,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":992703,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1538,"ondisk_log_size":1538,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6314912,"kb_used_data":11100,"kb_used_omap":12,"kb_used_meta":6303731,"kb_avail":743646304,"statfs":{"total":767960285184,"available":761493815296,"internally_reserved":0,"allocated":11366400,"data_stored":7478346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":6455021105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.012306"},"pg_stats":[{"pgid":"4.c","version":"421'30","reported_seq":105,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577091+0000","last_change":"2024-03-24T11:41:34.247353+0000","last_active":"2024-03-24T11:42:59.577091+0000","last_peered":"2024-03-24T11:42:59.577091+0000","last_clean":"2024-03-24T11:42:59.577091+0000","last_became_active":"2024-03-24T11:41:20.439720+0000","last_became_peered":"2024-03-24T11:41:20.439720+0000","last_unstale":"2024-03-24T11:42:59.577091+0000","last_undegraded":"2024-03-24T11:42:59.577091+0000","last_fullsized":"2024-03-24T11:42:59.577091+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:41:34.247251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:34.247251+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:03.233943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":980,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404035+0000","last_change":"2024-03-24T11:41:59.059226+0000","last_active":"2024-03-24T11:42:59.404035+0000","last_peered":"2024-03-24T11:42:59.404035+0000","last_clean":"2024-03-24T11:42:59.404035+0000","last_became_active":"2024-03-24T11:41:20.435862+0000","last_became_peered":"2024-03-24T11:41:20.435862+0000","last_unstale":"2024-03-24T11:42:59.404035+0000","last_undegraded":"2024-03-24T11:42:59.404035+0000","last_fullsized":"2024-03-24T11:42:59.404035+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:59.059120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:59.059120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:17.697948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"469'11","reported_seq":1337,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405259+0000","last_change":"2024-03-24T11:42:18.337741+0000","last_active":"2024-03-24T11:42:59.405259+0000","last_peered":"2024-03-24T11:42:59.405259+0000","last_clean":"2024-03-24T11:42:59.405259+0000","last_became_active":"2024-03-24T11:41:12.169429+0000","last_became_peered":"2024-03-24T11:41:12.169429+0000","last_unstale":"2024-03-24T11:42:59.405259+0000","last_undegraded":"2024-03-24T11:42:59.405259+0000","last_fullsized":"2024-03-24T11:42:59.405259+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'11","last_scrub_stamp":"2024-03-24T11:42:18.337621+0000","last_deep_scrub":"362'9","last_deep_scrub_stamp":"2024-03-24T11:35:58.633956+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.337621+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:43.657961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577444+0000","last_change":"2024-03-24T11:42:40.923244+0000","last_active":"2024-03-24T11:42:59.577444+0000","last_peered":"2024-03-24T11:42:59.577444+0000","last_clean":"2024-03-24T11:42:59.577444+0000","last_became_active":"2024-03-24T11:41:20.429395+0000","last_became_peered":"2024-03-24T11:41:20.429395+0000","last_unstale":"2024-03-24T11:42:59.577444+0000","last_undegraded":"2024-03-24T11:42:59.577444+0000","last_fullsized":"2024-03-24T11:42:59.577444+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:40.923126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:40.923126+0000","last_clean_scrub_stamp":"2024-03-24T11:42:40.923126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:52.913433+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":992,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013225+0000","last_change":"2024-03-24T11:42:47.987021+0000","last_active":"2024-03-24T11:43:00.013225+0000","last_peered":"2024-03-24T11:43:00.013225+0000","last_clean":"2024-03-24T11:43:00.013225+0000","last_became_active":"2024-03-24T11:42:23.704216+0000","last_became_peered":"2024-03-24T11:42:23.704216+0000","last_unstale":"2024-03-24T11:43:00.013225+0000","last_undegraded":"2024-03-24T11:43:00.013225+0000","last_fullsized":"2024-03-24T11:43:00.013225+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:47.986921+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:41:16.105950+0000","last_clean_scrub_stamp":"2024-03-24T11:42:47.986921+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.485326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"421'53","reported_seq":111,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577501+0000","last_change":"2024-03-24T11:42:13.131062+0000","last_active":"2024-03-24T11:42:59.577501+0000","last_peered":"2024-03-24T11:42:59.577501+0000","last_clean":"2024-03-24T11:42:59.577501+0000","last_became_active":"2024-03-24T11:41:20.434934+0000","last_became_peered":"2024-03-24T11:41:20.434934+0000","last_unstale":"2024-03-24T11:42:59.577501+0000","last_undegraded":"2024-03-24T11:42:59.577501+0000","last_fullsized":"2024-03-24T11:42:59.577501+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'53","last_scrub_stamp":"2024-03-24T11:42:13.130932+0000","last_deep_scrub":"213'39","last_deep_scrub_stamp":"2024-03-24T11:30:21.916549+0000","last_clean_scrub_stamp":"2024-03-24T11:42:13.130932+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:41.288616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"234'2","reported_seq":1045,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976559+0000","last_change":"2024-03-24T11:42:15.407993+0000","last_active":"2024-03-24T11:42:59.976559+0000","last_peered":"2024-03-24T11:42:59.976559+0000","last_clean":"2024-03-24T11:42:59.976559+0000","last_became_active":"2024-03-24T11:41:20.434125+0000","last_became_peered":"2024-03-24T11:41:20.434125+0000","last_unstale":"2024-03-24T11:42:59.976559+0000","last_undegraded":"2024-03-24T11:42:59.976559+0000","last_fullsized":"2024-03-24T11:42:59.976559+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'2","last_scrub_stamp":"2024-03-24T11:42:15.407821+0000","last_deep_scrub":"234'2","last_deep_scrub_stamp":"2024-03-24T11:30:19.680639+0000","last_clean_scrub_stamp":"2024-03-24T11:42:15.407821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:35.796023+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1013,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.268527+0000","last_change":"2024-03-24T11:43:00.268546+0000","last_active":"2024-03-24T11:43:00.268527+0000","last_peered":"2024-03-24T11:43:00.268527+0000","last_clean":"2024-03-24T11:43:00.268527+0000","last_became_active":"2024-03-24T11:41:20.434444+0000","last_became_peered":"2024-03-24T11:41:20.434444+0000","last_unstale":"2024-03-24T11:43:00.268527+0000","last_undegraded":"2024-03-24T11:43:00.268527+0000","last_fullsized":"2024-03-24T11:43:00.268527+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:00.268472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:18.788516+0000","last_clean_scrub_stamp":"2024-03-24T11:43:00.268472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:09.297578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":771,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405024+0000","last_change":"2024-03-24T11:42:01.492944+0000","last_active":"2024-03-24T11:42:59.405024+0000","last_peered":"2024-03-24T11:42:59.405024+0000","last_clean":"2024-03-24T11:42:59.405024+0000","last_became_active":"2024-03-24T11:41:49.149566+0000","last_became_peered":"2024-03-24T11:41:49.149566+0000","last_unstale":"2024-03-24T11:42:59.405024+0000","last_undegraded":"2024-03-24T11:42:59.405024+0000","last_fullsized":"2024-03-24T11:42:59.405024+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:01.492838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:01.492838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:07.335903+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"164'129","reported_seq":1159,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401662+0000","last_change":"2024-03-24T11:42:36.345577+0000","last_active":"2024-03-24T11:42:59.401662+0000","last_peered":"2024-03-24T11:42:59.401662+0000","last_clean":"2024-03-24T11:42:59.401662+0000","last_became_active":"2024-03-24T11:24:29.101590+0000","last_became_peered":"2024-03-24T11:24:29.101590+0000","last_unstale":"2024-03-24T11:42:59.401662+0000","last_undegraded":"2024-03-24T11:42:59.401662+0000","last_fullsized":"2024-03-24T11:42:59.401662+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"164'129","last_scrub_stamp":"2024-03-24T11:42:36.345473+0000","last_deep_scrub":"164'129","last_deep_scrub_stamp":"2024-03-24T11:31:31.621646+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.345473+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:54.986531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"421'27","reported_seq":105,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577253+0000","last_change":"2024-03-24T11:42:02.119690+0000","last_active":"2024-03-24T11:42:59.577253+0000","last_peered":"2024-03-24T11:42:59.577253+0000","last_clean":"2024-03-24T11:42:59.577253+0000","last_became_active":"2024-03-24T11:41:20.433184+0000","last_became_peered":"2024-03-24T11:41:20.433184+0000","last_unstale":"2024-03-24T11:42:59.577253+0000","last_undegraded":"2024-03-24T11:42:59.577253+0000","last_fullsized":"2024-03-24T11:42:59.577253+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:42:02.119468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:02.119468+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:24.522485+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576798+0000","last_change":"2024-03-24T11:42:18.038800+0000","last_active":"2024-03-24T11:42:59.576798+0000","last_peered":"2024-03-24T11:42:59.576798+0000","last_clean":"2024-03-24T11:42:59.576798+0000","last_became_active":"2024-03-24T11:41:20.431249+0000","last_became_peered":"2024-03-24T11:41:20.431249+0000","last_unstale":"2024-03-24T11:42:59.576798+0000","last_undegraded":"2024-03-24T11:42:59.576798+0000","last_fullsized":"2024-03-24T11:42:59.576798+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:18.038677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.038677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:28.393071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":997,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402711+0000","last_change":"2024-03-24T11:42:22.254996+0000","last_active":"2024-03-24T11:42:59.402711+0000","last_peered":"2024-03-24T11:42:59.402711+0000","last_clean":"2024-03-24T11:42:59.402711+0000","last_became_active":"2024-03-24T11:41:20.423777+0000","last_became_peered":"2024-03-24T11:41:20.423777+0000","last_unstale":"2024-03-24T11:42:59.402711+0000","last_undegraded":"2024-03-24T11:42:59.402711+0000","last_fullsized":"2024-03-24T11:42:59.402711+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:22.254896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:28.969611+0000","last_clean_scrub_stamp":"2024-03-24T11:42:22.254896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:35.361462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":956,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404155+0000","last_change":"2024-03-24T11:42:08.920820+0000","last_active":"2024-03-24T11:42:59.404155+0000","last_peered":"2024-03-24T11:42:59.404155+0000","last_clean":"2024-03-24T11:42:59.404155+0000","last_became_active":"2024-03-24T11:19:00.312122+0000","last_became_peered":"2024-03-24T11:19:00.312122+0000","last_unstale":"2024-03-24T11:42:59.404155+0000","last_undegraded":"2024-03-24T11:42:59.404155+0000","last_fullsized":"2024-03-24T11:42:59.404155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:08.920744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:08.920744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:22.522446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"421'46","reported_seq":1161,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:05.380518+0000","last_change":"2024-03-24T11:43:05.380555+0000","last_active":"2024-03-24T11:43:05.380518+0000","last_peered":"2024-03-24T11:43:05.380518+0000","last_clean":"2024-03-24T11:43:05.380518+0000","last_became_active":"2024-03-24T11:41:20.435397+0000","last_became_peered":"2024-03-24T11:41:20.435397+0000","last_unstale":"2024-03-24T11:43:05.380518+0000","last_undegraded":"2024-03-24T11:43:05.380518+0000","last_fullsized":"2024-03-24T11:43:05.380518+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-24T11:43:05.380455+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-03-24T11:20:32.534693+0000","last_clean_scrub_stamp":"2024-03-24T11:43:05.380455+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:10.826139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":989,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403414+0000","last_change":"2024-03-24T11:42:18.843034+0000","last_active":"2024-03-24T11:42:59.403414+0000","last_peered":"2024-03-24T11:42:59.403414+0000","last_clean":"2024-03-24T11:42:59.403414+0000","last_became_active":"2024-03-24T11:41:12.177833+0000","last_became_peered":"2024-03-24T11:41:12.177833+0000","last_unstale":"2024-03-24T11:42:59.403414+0000","last_undegraded":"2024-03-24T11:42:59.403414+0000","last_fullsized":"2024-03-24T11:42:59.403414+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:18.842953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:01.581731+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.842953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:27.663545+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"477'10","reported_seq":126,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577382+0000","last_change":"2024-03-24T11:42:51.937833+0000","last_active":"2024-03-24T11:42:59.577382+0000","last_peered":"2024-03-24T11:42:59.577382+0000","last_clean":"2024-03-24T11:42:59.577382+0000","last_became_active":"2024-03-24T11:41:20.443531+0000","last_became_peered":"2024-03-24T11:41:20.443531+0000","last_unstale":"2024-03-24T11:42:59.577382+0000","last_undegraded":"2024-03-24T11:42:59.577382+0000","last_fullsized":"2024-03-24T11:42:59.577382+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'10","last_scrub_stamp":"2024-03-24T11:42:51.937705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:51.937705+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.912013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":981,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:08.380817+0000","last_change":"2024-03-24T11:43:08.380844+0000","last_active":"2024-03-24T11:43:08.380817+0000","last_peered":"2024-03-24T11:43:08.380817+0000","last_clean":"2024-03-24T11:43:08.380817+0000","last_became_active":"2024-03-24T11:18:59.715142+0000","last_became_peered":"2024-03-24T11:18:59.715142+0000","last_unstale":"2024-03-24T11:43:08.380817+0000","last_undegraded":"2024-03-24T11:43:08.380817+0000","last_fullsized":"2024-03-24T11:43:08.380817+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:08.380706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:08.380706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:24.093442+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":1150,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401616+0000","last_change":"2024-03-24T11:42:32.355252+0000","last_active":"2024-03-24T11:42:59.401616+0000","last_peered":"2024-03-24T11:42:59.401616+0000","last_clean":"2024-03-24T11:42:59.401616+0000","last_became_active":"2024-03-24T11:24:29.083814+0000","last_became_peered":"2024-03-24T11:24:29.083814+0000","last_unstale":"2024-03-24T11:42:59.401616+0000","last_undegraded":"2024-03-24T11:42:59.401616+0000","last_fullsized":"2024-03-24T11:42:59.401616+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:32.355115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:42:32.355115+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:01.448797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576527+0000","last_change":"2024-03-24T11:42:36.858444+0000","last_active":"2024-03-24T11:42:59.576527+0000","last_peered":"2024-03-24T11:42:59.576527+0000","last_clean":"2024-03-24T11:42:59.576527+0000","last_became_active":"2024-03-24T11:41:20.439996+0000","last_became_peered":"2024-03-24T11:41:20.439996+0000","last_unstale":"2024-03-24T11:42:59.576527+0000","last_undegraded":"2024-03-24T11:42:59.576527+0000","last_fullsized":"2024-03-24T11:42:59.576527+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:36.858349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:32:39.715759+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.858349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.216884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'50","reported_seq":1082,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403454+0000","last_change":"2024-03-24T11:41:33.298242+0000","last_active":"2024-03-24T11:42:59.403454+0000","last_peered":"2024-03-24T11:42:59.403454+0000","last_clean":"2024-03-24T11:42:59.403454+0000","last_became_active":"2024-03-24T11:41:12.178352+0000","last_became_peered":"2024-03-24T11:41:12.178352+0000","last_unstale":"2024-03-24T11:42:59.403454+0000","last_undegraded":"2024-03-24T11:42:59.403454+0000","last_fullsized":"2024-03-24T11:42:59.403454+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'50","last_scrub_stamp":"2024-03-24T11:41:33.298084+0000","last_deep_scrub":"479'50","last_deep_scrub_stamp":"2024-03-24T11:41:33.298084+0000","last_clean_scrub_stamp":"2024-03-24T11:41:33.298084+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:03.215517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":981,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404563+0000","last_change":"2024-03-24T11:41:53.167160+0000","last_active":"2024-03-24T11:42:59.404563+0000","last_peered":"2024-03-24T11:42:59.404563+0000","last_clean":"2024-03-24T11:42:59.404563+0000","last_became_active":"2024-03-24T11:41:12.177324+0000","last_became_peered":"2024-03-24T11:41:12.177324+0000","last_unstale":"2024-03-24T11:42:59.404563+0000","last_undegraded":"2024-03-24T11:42:59.404563+0000","last_fullsized":"2024-03-24T11:42:59.404563+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:53.167086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:36.491006+0000","last_clean_scrub_stamp":"2024-03-24T11:41:53.167086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:07.262396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576673+0000","last_change":"2024-03-24T11:42:11.067151+0000","last_active":"2024-03-24T11:42:59.576673+0000","last_peered":"2024-03-24T11:42:59.576673+0000","last_clean":"2024-03-24T11:42:59.576673+0000","last_became_active":"2024-03-24T11:41:20.441367+0000","last_became_peered":"2024-03-24T11:41:20.441367+0000","last_unstale":"2024-03-24T11:42:59.576673+0000","last_undegraded":"2024-03-24T11:42:59.576673+0000","last_fullsized":"2024-03-24T11:42:59.576673+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:11.067050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:17.941259+0000","last_clean_scrub_stamp":"2024-03-24T11:42:11.067050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.818522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":618,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577504+0000","last_change":"2024-03-24T11:42:22.990318+0000","last_active":"2024-03-24T11:42:59.577504+0000","last_peered":"2024-03-24T11:42:59.577504+0000","last_clean":"2024-03-24T11:42:59.577504+0000","last_became_active":"2024-03-24T11:41:52.546134+0000","last_became_peered":"2024-03-24T11:41:52.546134+0000","last_unstale":"2024-03-24T11:42:59.577504+0000","last_undegraded":"2024-03-24T11:42:59.577504+0000","last_fullsized":"2024-03-24T11:42:59.577504+0000","mapping_epoch":487,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:22.990174+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:36:59.739800+0000","last_clean_scrub_stamp":"2024-03-24T11:42:22.990174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:30.365210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"477'10","reported_seq":1282,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:05.477671+0000","last_change":"2024-03-24T11:42:40.604076+0000","last_active":"2024-03-24T11:43:05.477671+0000","last_peered":"2024-03-24T11:43:05.477671+0000","last_clean":"2024-03-24T11:43:05.477671+0000","last_became_active":"2024-03-24T11:41:20.422150+0000","last_became_peered":"2024-03-24T11:41:20.422150+0000","last_unstale":"2024-03-24T11:43:05.477671+0000","last_undegraded":"2024-03-24T11:43:05.477671+0000","last_fullsized":"2024-03-24T11:43:05.477671+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'10","last_scrub_stamp":"2024-03-24T11:42:40.603935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:40.603935+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.252161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"421'42","reported_seq":117,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576659+0000","last_change":"2024-03-24T11:42:59.067343+0000","last_active":"2024-03-24T11:42:59.576659+0000","last_peered":"2024-03-24T11:42:59.576659+0000","last_clean":"2024-03-24T11:42:59.576659+0000","last_became_active":"2024-03-24T11:41:20.436237+0000","last_became_peered":"2024-03-24T11:41:20.436237+0000","last_unstale":"2024-03-24T11:42:59.576659+0000","last_undegraded":"2024-03-24T11:42:59.576659+0000","last_fullsized":"2024-03-24T11:42:59.576659+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'42","last_scrub_stamp":"2024-03-24T11:42:59.067206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:59.067206+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:17.848043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":1043,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013952+0000","last_change":"2024-03-24T11:42:14.441001+0000","last_active":"2024-03-24T11:43:00.013952+0000","last_peered":"2024-03-24T11:43:00.013952+0000","last_clean":"2024-03-24T11:43:00.013952+0000","last_became_active":"2024-03-24T11:41:20.432418+0000","last_became_peered":"2024-03-24T11:41:20.432418+0000","last_unstale":"2024-03-24T11:43:00.013952+0000","last_undegraded":"2024-03-24T11:43:00.013952+0000","last_fullsized":"2024-03-24T11:43:00.013952+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:42:14.440869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:14.440869+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:18.772643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":550,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013496+0000","last_change":"2024-03-24T11:42:23.706157+0000","last_active":"2024-03-24T11:43:00.013496+0000","last_peered":"2024-03-24T11:43:00.013496+0000","last_clean":"2024-03-24T11:43:00.013496+0000","last_became_active":"2024-03-24T11:42:23.704839+0000","last_became_peered":"2024-03-24T11:42:23.704839+0000","last_unstale":"2024-03-24T11:43:00.013496+0000","last_undegraded":"2024-03-24T11:43:00.013496+0000","last_fullsized":"2024-03-24T11:43:00.013496+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:13.448157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:13.448157+0000","last_clean_scrub_stamp":"2024-03-24T11:42:13.448157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:39.386341+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1192,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404009+0000","last_change":"2024-03-24T11:41:43.186226+0000","last_active":"2024-03-24T11:42:59.404009+0000","last_peered":"2024-03-24T11:42:59.404009+0000","last_clean":"2024-03-24T11:42:59.404009+0000","last_became_active":"2024-03-24T11:41:15.157694+0000","last_became_peered":"2024-03-24T11:41:15.157694+0000","last_unstale":"2024-03-24T11:42:59.404009+0000","last_undegraded":"2024-03-24T11:42:59.404009+0000","last_fullsized":"2024-03-24T11:42:59.404009+0000","mapping_epoch":470,"log_start":"83'128","ondisk_log_start":"83'128","created":199,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:41:43.186127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:41:43.186127+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:12.002602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"186'9","reported_seq":1294,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013335+0000","last_change":"2024-03-24T11:42:29.519698+0000","last_active":"2024-03-24T11:43:00.013335+0000","last_peered":"2024-03-24T11:43:00.013335+0000","last_clean":"2024-03-24T11:43:00.013335+0000","last_became_active":"2024-03-24T11:18:57.697043+0000","last_became_peered":"2024-03-24T11:18:57.697043+0000","last_unstale":"2024-03-24T11:43:00.013335+0000","last_undegraded":"2024-03-24T11:43:00.013335+0000","last_fullsized":"2024-03-24T11:43:00.013335+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'9","last_scrub_stamp":"2024-03-24T11:42:29.519427+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:41:03.262723+0000","last_clean_scrub_stamp":"2024-03-24T11:42:29.519427+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:40.460395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"421'30","reported_seq":1079,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013710+0000","last_change":"2024-03-24T11:41:43.021116+0000","last_active":"2024-03-24T11:43:00.013710+0000","last_peered":"2024-03-24T11:43:00.013710+0000","last_clean":"2024-03-24T11:43:00.013710+0000","last_became_active":"2024-03-24T11:41:20.438781+0000","last_became_peered":"2024-03-24T11:41:20.438781+0000","last_unstale":"2024-03-24T11:43:00.013710+0000","last_undegraded":"2024-03-24T11:43:00.013710+0000","last_fullsized":"2024-03-24T11:43:00.013710+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:41:43.021020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:43.021020+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:11.038809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":142,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976841+0000","last_change":"2024-03-24T11:42:28.313341+0000","last_active":"2024-03-24T11:42:59.976841+0000","last_peered":"2024-03-24T11:42:59.976841+0000","last_clean":"2024-03-24T11:42:59.976841+0000","last_became_active":"2024-03-24T11:40:35.291074+0000","last_became_peered":"2024-03-24T11:40:35.291074+0000","last_unstale":"2024-03-24T11:42:59.976841+0000","last_undegraded":"2024-03-24T11:42:59.976841+0000","last_fullsized":"2024-03-24T11:42:59.976841+0000","mapping_epoch":454,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:28.313240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:11.388851+0000","last_clean_scrub_stamp":"2024-03-24T11:42:28.313240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:36.809012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"73'1","reported_seq":984,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012852+0000","last_change":"2024-03-24T11:42:15.523777+0000","last_active":"2024-03-24T11:43:00.012852+0000","last_peered":"2024-03-24T11:43:00.012852+0000","last_clean":"2024-03-24T11:43:00.012852+0000","last_became_active":"2024-03-24T11:18:59.714560+0000","last_became_peered":"2024-03-24T11:18:59.714560+0000","last_unstale":"2024-03-24T11:43:00.012852+0000","last_undegraded":"2024-03-24T11:43:00.012852+0000","last_fullsized":"2024-03-24T11:43:00.012852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-03-24T11:42:15.523653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:15.523653+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:21.046051+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1229,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977157+0000","last_change":"2024-03-24T11:42:45.335848+0000","last_active":"2024-03-24T11:42:59.977157+0000","last_peered":"2024-03-24T11:42:59.977157+0000","last_clean":"2024-03-24T11:42:59.977157+0000","last_became_active":"2024-03-24T11:41:44.506953+0000","last_became_peered":"2024-03-24T11:41:44.506953+0000","last_unstale":"2024-03-24T11:42:59.977157+0000","last_undegraded":"2024-03-24T11:42:59.977157+0000","last_fullsized":"2024-03-24T11:42:59.977157+0000","mapping_epoch":482,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:45.335735+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:42:45.335735+0000","last_clean_scrub_stamp":"2024-03-24T11:42:45.335735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.080281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1018,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976945+0000","last_change":"2024-03-24T11:42:35.366426+0000","last_active":"2024-03-24T11:42:59.976945+0000","last_peered":"2024-03-24T11:42:59.976945+0000","last_clean":"2024-03-24T11:42:59.976945+0000","last_became_active":"2024-03-24T11:18:57.698564+0000","last_became_peered":"2024-03-24T11:18:57.698564+0000","last_unstale":"2024-03-24T11:42:59.976945+0000","last_undegraded":"2024-03-24T11:42:59.976945+0000","last_fullsized":"2024-03-24T11:42:59.976945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:35.366331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:35.366331+0000","last_clean_scrub_stamp":"2024-03-24T11:42:35.366331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.513525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"421'39","reported_seq":1112,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012989+0000","last_change":"2024-03-24T11:42:48.683947+0000","last_active":"2024-03-24T11:43:00.012989+0000","last_peered":"2024-03-24T11:43:00.012989+0000","last_clean":"2024-03-24T11:43:00.012989+0000","last_became_active":"2024-03-24T11:41:12.177000+0000","last_became_peered":"2024-03-24T11:41:12.177000+0000","last_unstale":"2024-03-24T11:43:00.012989+0000","last_undegraded":"2024-03-24T11:43:00.012989+0000","last_fullsized":"2024-03-24T11:43:00.012989+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:42:48.683793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:48.683793+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:10.986638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1010,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013716+0000","last_change":"2024-03-24T11:42:50.129608+0000","last_active":"2024-03-24T11:43:00.013716+0000","last_peered":"2024-03-24T11:43:00.013716+0000","last_clean":"2024-03-24T11:43:00.013716+0000","last_became_active":"2024-03-24T11:18:53.676999+0000","last_became_peered":"2024-03-24T11:18:53.676999+0000","last_unstale":"2024-03-24T11:43:00.013716+0000","last_undegraded":"2024-03-24T11:43:00.013716+0000","last_fullsized":"2024-03-24T11:43:00.013716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:50.129525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:50.129525+0000","last_clean_scrub_stamp":"2024-03-24T11:42:50.129525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:17.910565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":990,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012875+0000","last_change":"2024-03-24T11:42:38.080747+0000","last_active":"2024-03-24T11:43:00.012875+0000","last_peered":"2024-03-24T11:43:00.012875+0000","last_clean":"2024-03-24T11:43:00.012875+0000","last_became_active":"2024-03-24T11:19:00.311632+0000","last_became_peered":"2024-03-24T11:19:00.311632+0000","last_unstale":"2024-03-24T11:43:00.012875+0000","last_undegraded":"2024-03-24T11:43:00.012875+0000","last_fullsized":"2024-03-24T11:43:00.012875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:38.080629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:38.080629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:43.603608+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1195,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977166+0000","last_change":"2024-03-24T11:42:36.319294+0000","last_active":"2024-03-24T11:42:59.977166+0000","last_peered":"2024-03-24T11:42:59.977166+0000","last_clean":"2024-03-24T11:42:59.977166+0000","last_became_active":"2024-03-24T11:35:43.803294+0000","last_became_peered":"2024-03-24T11:35:43.803294+0000","last_unstale":"2024-03-24T11:42:59.977166+0000","last_undegraded":"2024-03-24T11:42:59.977166+0000","last_fullsized":"2024-03-24T11:42:59.977166+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:36.319154+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:39:14.270084+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.319154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:54.924573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":986,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402780+0000","last_change":"2024-03-24T11:41:47.050109+0000","last_active":"2024-03-24T11:42:59.402780+0000","last_peered":"2024-03-24T11:42:59.402780+0000","last_clean":"2024-03-24T11:42:59.402780+0000","last_became_active":"2024-03-24T11:18:57.692813+0000","last_became_peered":"2024-03-24T11:18:57.692813+0000","last_unstale":"2024-03-24T11:42:59.402780+0000","last_undegraded":"2024-03-24T11:42:59.402780+0000","last_fullsized":"2024-03-24T11:42:59.402780+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:47.049971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:30:27.391607+0000","last_clean_scrub_stamp":"2024-03-24T11:41:47.049971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:07.661474+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"421'36","reported_seq":553,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402292+0000","last_change":"2024-03-24T11:41:42.007388+0000","last_active":"2024-03-24T11:42:59.402292+0000","last_peered":"2024-03-24T11:42:59.402292+0000","last_clean":"2024-03-24T11:42:59.402292+0000","last_became_active":"2024-03-24T11:33:10.451794+0000","last_became_peered":"2024-03-24T11:33:10.451794+0000","last_unstale":"2024-03-24T11:42:59.402292+0000","last_undegraded":"2024-03-24T11:42:59.402292+0000","last_fullsized":"2024-03-24T11:42:59.402292+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'36","last_scrub_stamp":"2024-03-24T11:41:42.007263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:42.007263+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:09.419440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1276,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:06.954092+0000","last_change":"2024-03-24T11:43:06.954092+0000","last_active":"2024-03-24T11:43:06.954092+0000","last_peered":"2024-03-24T11:43:06.954092+0000","last_clean":"2024-03-24T11:43:06.954092+0000","last_became_active":"2024-03-24T11:42:23.696556+0000","last_became_peered":"2024-03-24T11:42:23.696556+0000","last_unstale":"2024-03-24T11:43:06.954092+0000","last_undegraded":"2024-03-24T11:43:06.954092+0000","last_fullsized":"2024-03-24T11:43:06.954092+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T11:43:06.953977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:43:06.953977+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:22.128458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075615899999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":675,"num_read_kb":670,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1017,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012709+0000","last_change":"2024-03-24T11:42:34.573788+0000","last_active":"2024-03-24T11:43:00.012709+0000","last_peered":"2024-03-24T11:43:00.012709+0000","last_clean":"2024-03-24T11:43:00.012709+0000","last_became_active":"2024-03-24T11:41:20.423214+0000","last_became_peered":"2024-03-24T11:41:20.423214+0000","last_unstale":"2024-03-24T11:43:00.012709+0000","last_undegraded":"2024-03-24T11:43:00.012709+0000","last_fullsized":"2024-03-24T11:43:00.012709+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:34.573666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:34.573666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:38.297826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":998,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013628+0000","last_change":"2024-03-24T11:42:54.144105+0000","last_active":"2024-03-24T11:43:00.013628+0000","last_peered":"2024-03-24T11:43:00.013628+0000","last_clean":"2024-03-24T11:43:00.013628+0000","last_became_active":"2024-03-24T11:18:59.710413+0000","last_became_peered":"2024-03-24T11:18:59.710413+0000","last_unstale":"2024-03-24T11:43:00.013628+0000","last_undegraded":"2024-03-24T11:43:00.013628+0000","last_fullsized":"2024-03-24T11:43:00.013628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:54.144022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:54.144022+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.144022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:07.489551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1217,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013401+0000","last_change":"2024-03-24T11:41:53.097361+0000","last_active":"2024-03-24T11:43:00.013401+0000","last_peered":"2024-03-24T11:43:00.013401+0000","last_clean":"2024-03-24T11:43:00.013401+0000","last_became_active":"2024-03-24T11:41:52.545645+0000","last_became_peered":"2024-03-24T11:41:52.545645+0000","last_unstale":"2024-03-24T11:43:00.013401+0000","last_undegraded":"2024-03-24T11:43:00.013401+0000","last_fullsized":"2024-03-24T11:43:00.013401+0000","mapping_epoch":487,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:41:53.097276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:41:53.097276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:14.580832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":558,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013670+0000","last_change":"2024-03-24T11:42:57.897663+0000","last_active":"2024-03-24T11:43:00.013670+0000","last_peered":"2024-03-24T11:43:00.013670+0000","last_clean":"2024-03-24T11:43:00.013670+0000","last_became_active":"2024-03-24T11:42:23.704051+0000","last_became_peered":"2024-03-24T11:42:23.704051+0000","last_unstale":"2024-03-24T11:43:00.013670+0000","last_undegraded":"2024-03-24T11:43:00.013670+0000","last_fullsized":"2024-03-24T11:43:00.013670+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:57.897437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:40.290398+0000","last_clean_scrub_stamp":"2024-03-24T11:42:57.897437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.930802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"477'52","reported_seq":1467,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977094+0000","last_change":"2024-03-24T11:42:10.342514+0000","last_active":"2024-03-24T11:42:59.977094+0000","last_peered":"2024-03-24T11:42:59.977094+0000","last_clean":"2024-03-24T11:42:59.977094+0000","last_became_active":"2024-03-24T11:41:20.444994+0000","last_became_peered":"2024-03-24T11:41:20.444994+0000","last_unstale":"2024-03-24T11:42:59.977094+0000","last_undegraded":"2024-03-24T11:42:59.977094+0000","last_fullsized":"2024-03-24T11:42:59.977094+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'52","last_scrub_stamp":"2024-03-24T11:42:10.342346+0000","last_deep_scrub":"176'24","last_deep_scrub_stamp":"2024-03-24T11:26:37.676764+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.342346+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:40.038096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":981,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403821+0000","last_change":"2024-03-24T11:41:41.160247+0000","last_active":"2024-03-24T11:42:59.403821+0000","last_peered":"2024-03-24T11:42:59.403821+0000","last_clean":"2024-03-24T11:42:59.403821+0000","last_became_active":"2024-03-24T11:39:28.584724+0000","last_became_peered":"2024-03-24T11:39:28.584724+0000","last_unstale":"2024-03-24T11:42:59.403821+0000","last_undegraded":"2024-03-24T11:42:59.403821+0000","last_fullsized":"2024-03-24T11:42:59.403821+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:41.160141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:41.160141+0000","last_clean_scrub_stamp":"2024-03-24T11:41:41.160141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":968,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013016+0000","last_change":"2024-03-24T11:42:30.879238+0000","last_active":"2024-03-24T11:43:00.013016+0000","last_peered":"2024-03-24T11:43:00.013016+0000","last_clean":"2024-03-24T11:43:00.013016+0000","last_became_active":"2024-03-24T11:42:23.700644+0000","last_became_peered":"2024-03-24T11:42:23.700644+0000","last_unstale":"2024-03-24T11:43:00.013016+0000","last_undegraded":"2024-03-24T11:43:00.013016+0000","last_fullsized":"2024-03-24T11:43:00.013016+0000","mapping_epoch":502,"log_start":"83'128","ondisk_log_start":"83'128","created":20,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:30.879136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:42:30.879136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:41.591889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1010,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402422+0000","last_change":"2024-03-24T11:42:31.531147+0000","last_active":"2024-03-24T11:42:59.402422+0000","last_peered":"2024-03-24T11:42:59.402422+0000","last_clean":"2024-03-24T11:42:59.402422+0000","last_became_active":"2024-03-24T11:41:20.432083+0000","last_became_peered":"2024-03-24T11:41:20.432083+0000","last_unstale":"2024-03-24T11:42:59.402422+0000","last_undegraded":"2024-03-24T11:42:59.402422+0000","last_fullsized":"2024-03-24T11:42:59.402422+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:31.531021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:49.074485+0000","last_clean_scrub_stamp":"2024-03-24T11:42:31.531021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:35.817363+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1030,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404860+0000","last_change":"2024-03-24T11:42:27.325301+0000","last_active":"2024-03-24T11:42:59.404860+0000","last_peered":"2024-03-24T11:42:59.404860+0000","last_clean":"2024-03-24T11:42:59.404860+0000","last_became_active":"2024-03-24T11:41:12.170158+0000","last_became_peered":"2024-03-24T11:41:12.170158+0000","last_unstale":"2024-03-24T11:42:59.404860+0000","last_undegraded":"2024-03-24T11:42:59.404860+0000","last_fullsized":"2024-03-24T11:42:59.404860+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:27.325221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.325221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.090749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"449'26","reported_seq":1041,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:06.218666+0000","last_change":"2024-03-24T11:43:06.218666+0000","last_active":"2024-03-24T11:43:06.218666+0000","last_peered":"2024-03-24T11:43:06.218666+0000","last_clean":"2024-03-24T11:43:06.218666+0000","last_became_active":"2024-03-24T11:19:18.819406+0000","last_became_peered":"2024-03-24T11:19:18.819406+0000","last_unstale":"2024-03-24T11:43:06.218666+0000","last_undegraded":"2024-03-24T11:43:06.218666+0000","last_fullsized":"2024-03-24T11:43:06.218666+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'26","last_scrub_stamp":"2024-03-24T11:43:06.218556+0000","last_deep_scrub":"449'26","last_deep_scrub_stamp":"2024-03-24T11:40:22.408163+0000","last_clean_scrub_stamp":"2024-03-24T11:43:06.218556+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:34.213179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1035,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404295+0000","last_change":"2024-03-24T11:42:32.384269+0000","last_active":"2024-03-24T11:42:59.404295+0000","last_peered":"2024-03-24T11:42:59.404295+0000","last_clean":"2024-03-24T11:42:59.404295+0000","last_became_active":"2024-03-24T11:18:52.674812+0000","last_became_peered":"2024-03-24T11:18:52.674812+0000","last_unstale":"2024-03-24T11:42:59.404295+0000","last_undegraded":"2024-03-24T11:42:59.404295+0000","last_fullsized":"2024-03-24T11:42:59.404295+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:32.384181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:21.288390+0000","last_clean_scrub_stamp":"2024-03-24T11:42:32.384181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:47.116360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064702100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1011,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013165+0000","last_change":"2024-03-24T11:42:22.106589+0000","last_active":"2024-03-24T11:43:00.013165+0000","last_peered":"2024-03-24T11:43:00.013165+0000","last_clean":"2024-03-24T11:43:00.013165+0000","last_became_active":"2024-03-24T11:18:53.679830+0000","last_became_peered":"2024-03-24T11:18:53.679830+0000","last_unstale":"2024-03-24T11:43:00.013165+0000","last_undegraded":"2024-03-24T11:43:00.013165+0000","last_fullsized":"2024-03-24T11:43:00.013165+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:22.106466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:08.778916+0000","last_clean_scrub_stamp":"2024-03-24T11:42:22.106466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:27.333873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":550,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013983+0000","last_change":"2024-03-24T11:42:23.702546+0000","last_active":"2024-03-24T11:43:00.013983+0000","last_peered":"2024-03-24T11:43:00.013983+0000","last_clean":"2024-03-24T11:43:00.013983+0000","last_became_active":"2024-03-24T11:42:23.701790+0000","last_became_peered":"2024-03-24T11:42:23.701790+0000","last_unstale":"2024-03-24T11:43:00.013983+0000","last_undegraded":"2024-03-24T11:43:00.013983+0000","last_fullsized":"2024-03-24T11:43:00.013983+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:21.466092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:44.288152+0000","last_clean_scrub_stamp":"2024-03-24T11:42:21.466092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.644913+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":914,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013241+0000","last_change":"2024-03-24T11:42:23.700364+0000","last_active":"2024-03-24T11:43:00.013241+0000","last_peered":"2024-03-24T11:43:00.013241+0000","last_clean":"2024-03-24T11:43:00.013241+0000","last_became_active":"2024-03-24T11:42:23.700004+0000","last_became_peered":"2024-03-24T11:42:23.700004+0000","last_unstale":"2024-03-24T11:43:00.013241+0000","last_undegraded":"2024-03-24T11:43:00.013241+0000","last_fullsized":"2024-03-24T11:43:00.013241+0000","mapping_epoch":502,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:21.495309+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T11:27:40.000450+0000","last_clean_scrub_stamp":"2024-03-24T11:42:21.495309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:44.063958+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1006,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013359+0000","last_change":"2024-03-24T11:42:55.714126+0000","last_active":"2024-03-24T11:43:00.013359+0000","last_peered":"2024-03-24T11:43:00.013359+0000","last_clean":"2024-03-24T11:43:00.013359+0000","last_became_active":"2024-03-24T11:41:20.424238+0000","last_became_peered":"2024-03-24T11:41:20.424238+0000","last_unstale":"2024-03-24T11:43:00.013359+0000","last_undegraded":"2024-03-24T11:43:00.013359+0000","last_fullsized":"2024-03-24T11:43:00.013359+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:55.714040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:37.333168+0000","last_clean_scrub_stamp":"2024-03-24T11:42:55.714040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:00.006863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"421'53","reported_seq":1401,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013833+0000","last_change":"2024-03-24T11:42:25.133503+0000","last_active":"2024-03-24T11:43:00.013833+0000","last_peered":"2024-03-24T11:43:00.013833+0000","last_clean":"2024-03-24T11:43:00.013833+0000","last_became_active":"2024-03-24T11:37:33.888447+0000","last_became_peered":"2024-03-24T11:37:33.888447+0000","last_unstale":"2024-03-24T11:43:00.013833+0000","last_undegraded":"2024-03-24T11:43:00.013833+0000","last_fullsized":"2024-03-24T11:43:00.013833+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'53","last_scrub_stamp":"2024-03-24T11:42:25.133396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:25.133396+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.907131+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577033+0000","last_change":"2024-03-24T11:42:09.057007+0000","last_active":"2024-03-24T11:42:59.577033+0000","last_peered":"2024-03-24T11:42:59.577033+0000","last_clean":"2024-03-24T11:42:59.577033+0000","last_became_active":"2024-03-24T11:41:20.442409+0000","last_became_peered":"2024-03-24T11:41:20.442409+0000","last_unstale":"2024-03-24T11:42:59.577033+0000","last_undegraded":"2024-03-24T11:42:59.577033+0000","last_fullsized":"2024-03-24T11:42:59.577033+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:09.056911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:01.964189+0000","last_clean_scrub_stamp":"2024-03-24T11:42:09.056911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:33.861135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1013,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401808+0000","last_change":"2024-03-24T11:42:54.270181+0000","last_active":"2024-03-24T11:42:59.401808+0000","last_peered":"2024-03-24T11:42:59.401808+0000","last_clean":"2024-03-24T11:42:59.401808+0000","last_became_active":"2024-03-24T11:18:52.677773+0000","last_became_peered":"2024-03-24T11:18:52.677773+0000","last_unstale":"2024-03-24T11:42:59.401808+0000","last_undegraded":"2024-03-24T11:42:59.401808+0000","last_fullsized":"2024-03-24T11:42:59.401808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-24T11:42:54.270078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:51.653668+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.270078+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:05.891492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056060300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":989,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:02.475651+0000","last_change":"2024-03-24T11:43:02.475692+0000","last_active":"2024-03-24T11:43:02.475651+0000","last_peered":"2024-03-24T11:43:02.475651+0000","last_clean":"2024-03-24T11:43:02.475651+0000","last_became_active":"2024-03-24T11:41:20.437191+0000","last_became_peered":"2024-03-24T11:41:20.437191+0000","last_unstale":"2024-03-24T11:43:02.475651+0000","last_undegraded":"2024-03-24T11:43:02.475651+0000","last_fullsized":"2024-03-24T11:43:02.475651+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:02.475564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:02.475564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:31.938527+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1181,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401540+0000","last_change":"2024-03-24T11:42:10.200089+0000","last_active":"2024-03-24T11:42:59.401540+0000","last_peered":"2024-03-24T11:42:59.401540+0000","last_clean":"2024-03-24T11:42:59.401540+0000","last_became_active":"2024-03-24T11:35:31.681454+0000","last_became_peered":"2024-03-24T11:35:31.681454+0000","last_unstale":"2024-03-24T11:42:59.401540+0000","last_undegraded":"2024-03-24T11:42:59.401540+0000","last_fullsized":"2024-03-24T11:42:59.401540+0000","mapping_epoch":141,"log_start":"83'128","ondisk_log_start":"83'128","created":136,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T11:42:10.200004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:16:40.048291+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.200004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.825400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"421'21","reported_seq":1063,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013461+0000","last_change":"2024-03-24T11:42:36.575963+0000","last_active":"2024-03-24T11:43:00.013461+0000","last_peered":"2024-03-24T11:43:00.013461+0000","last_clean":"2024-03-24T11:43:00.013461+0000","last_became_active":"2024-03-24T11:41:20.431968+0000","last_became_peered":"2024-03-24T11:41:20.431968+0000","last_unstale":"2024-03-24T11:43:00.013461+0000","last_undegraded":"2024-03-24T11:43:00.013461+0000","last_fullsized":"2024-03-24T11:43:00.013461+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:36.575864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.575864+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.461109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"510'2","reported_seq":1008,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402237+0000","last_change":"2024-03-24T11:42:44.413123+0000","last_active":"2024-03-24T11:42:59.402237+0000","last_peered":"2024-03-24T11:42:59.402237+0000","last_clean":"2024-03-24T11:42:59.402237+0000","last_became_active":"2024-03-24T11:18:53.680470+0000","last_became_peered":"2024-03-24T11:18:53.680470+0000","last_unstale":"2024-03-24T11:42:59.402237+0000","last_undegraded":"2024-03-24T11:42:59.402237+0000","last_fullsized":"2024-03-24T11:42:59.402237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'2","last_scrub_stamp":"2024-03-24T11:42:44.412955+0000","last_deep_scrub":"510'2","last_deep_scrub_stamp":"2024-03-24T11:42:44.412955+0000","last_clean_scrub_stamp":"2024-03-24T11:42:44.412955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:52.845399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":776,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404643+0000","last_change":"2024-03-24T11:42:59.392439+0000","last_active":"2024-03-24T11:42:59.404643+0000","last_peered":"2024-03-24T11:42:59.404643+0000","last_clean":"2024-03-24T11:42:59.404643+0000","last_became_active":"2024-03-24T11:41:49.149359+0000","last_became_peered":"2024-03-24T11:41:49.149359+0000","last_unstale":"2024-03-24T11:42:59.404643+0000","last_undegraded":"2024-03-24T11:42:59.404643+0000","last_fullsized":"2024-03-24T11:42:59.404643+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:59.392349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:39.212274+0000","last_clean_scrub_stamp":"2024-03-24T11:42:59.392349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:21.897033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1002,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012880+0000","last_change":"2024-03-24T11:42:45.765441+0000","last_active":"2024-03-24T11:43:00.012880+0000","last_peered":"2024-03-24T11:43:00.012880+0000","last_clean":"2024-03-24T11:43:00.012880+0000","last_became_active":"2024-03-24T11:41:12.168488+0000","last_became_peered":"2024-03-24T11:41:12.168488+0000","last_unstale":"2024-03-24T11:43:00.012880+0000","last_undegraded":"2024-03-24T11:43:00.012880+0000","last_fullsized":"2024-03-24T11:43:00.012880+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:45.765340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:32.567864+0000","last_clean_scrub_stamp":"2024-03-24T11:42:45.765340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.324863+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"421'60","reported_seq":1154,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976774+0000","last_change":"2024-03-24T11:42:57.287304+0000","last_active":"2024-03-24T11:42:59.976774+0000","last_peered":"2024-03-24T11:42:59.976774+0000","last_clean":"2024-03-24T11:42:59.976774+0000","last_became_active":"2024-03-24T11:41:12.181243+0000","last_became_peered":"2024-03-24T11:41:12.181243+0000","last_unstale":"2024-03-24T11:42:59.976774+0000","last_undegraded":"2024-03-24T11:42:59.976774+0000","last_fullsized":"2024-03-24T11:42:59.976774+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'60","last_scrub_stamp":"2024-03-24T11:42:57.287151+0000","last_deep_scrub":"421'60","last_deep_scrub_stamp":"2024-03-24T11:41:40.343360+0000","last_clean_scrub_stamp":"2024-03-24T11:42:57.287151+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:16.888131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":547,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013737+0000","last_change":"2024-03-24T11:42:47.008473+0000","last_active":"2024-03-24T11:43:00.013737+0000","last_peered":"2024-03-24T11:43:00.013737+0000","last_clean":"2024-03-24T11:43:00.013737+0000","last_became_active":"2024-03-24T11:42:23.701892+0000","last_became_peered":"2024-03-24T11:42:23.701892+0000","last_unstale":"2024-03-24T11:43:00.013737+0000","last_undegraded":"2024-03-24T11:43:00.013737+0000","last_fullsized":"2024-03-24T11:43:00.013737+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:47.008366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:30.990378+0000","last_clean_scrub_stamp":"2024-03-24T11:42:47.008366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:16.212319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1017,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404977+0000","last_change":"2024-03-24T11:42:28.372972+0000","last_active":"2024-03-24T11:42:59.404977+0000","last_peered":"2024-03-24T11:42:59.404977+0000","last_clean":"2024-03-24T11:42:59.404977+0000","last_became_active":"2024-03-24T11:18:57.699551+0000","last_became_peered":"2024-03-24T11:18:57.699551+0000","last_unstale":"2024-03-24T11:42:59.404977+0000","last_undegraded":"2024-03-24T11:42:59.404977+0000","last_fullsized":"2024-03-24T11:42:59.404977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:28.372893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:27.894028+0000","last_clean_scrub_stamp":"2024-03-24T11:42:28.372893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:31.752046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":339,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013217+0000","last_change":"2024-03-24T11:42:23.705318+0000","last_active":"2024-03-24T11:43:00.013217+0000","last_peered":"2024-03-24T11:43:00.013217+0000","last_clean":"2024-03-24T11:43:00.013217+0000","last_became_active":"2024-03-24T11:42:23.704796+0000","last_became_peered":"2024-03-24T11:42:23.704796+0000","last_unstale":"2024-03-24T11:43:00.013217+0000","last_undegraded":"2024-03-24T11:43:00.013217+0000","last_fullsized":"2024-03-24T11:43:00.013217+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:55.379062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:38:12.123323+0000","last_clean_scrub_stamp":"2024-03-24T11:41:55.379062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:05.201594+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":784,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403571+0000","last_change":"2024-03-24T11:42:10.282085+0000","last_active":"2024-03-24T11:42:59.403571+0000","last_peered":"2024-03-24T11:42:59.403571+0000","last_clean":"2024-03-24T11:42:59.403571+0000","last_became_active":"2024-03-24T11:41:49.360640+0000","last_became_peered":"2024-03-24T11:41:49.360640+0000","last_unstale":"2024-03-24T11:42:59.403571+0000","last_undegraded":"2024-03-24T11:42:59.403571+0000","last_fullsized":"2024-03-24T11:42:59.403571+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:10.282008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.282008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:34.938772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"480'29","reported_seq":118,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576670+0000","last_change":"2024-03-24T11:42:56.057373+0000","last_active":"2024-03-24T11:42:59.576670+0000","last_peered":"2024-03-24T11:42:59.576670+0000","last_clean":"2024-03-24T11:42:59.576670+0000","last_became_active":"2024-03-24T11:41:20.441074+0000","last_became_peered":"2024-03-24T11:41:20.441074+0000","last_unstale":"2024-03-24T11:42:59.576670+0000","last_undegraded":"2024-03-24T11:42:59.576670+0000","last_fullsized":"2024-03-24T11:42:59.576670+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'29","last_scrub_stamp":"2024-03-24T11:42:56.057229+0000","last_deep_scrub":"480'29","last_deep_scrub_stamp":"2024-03-24T11:41:37.153709+0000","last_clean_scrub_stamp":"2024-03-24T11:42:56.057229+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:23.891977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":232,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013815+0000","last_change":"2024-03-24T11:42:26.847291+0000","last_active":"2024-03-24T11:43:00.013815+0000","last_peered":"2024-03-24T11:43:00.013815+0000","last_clean":"2024-03-24T11:43:00.013815+0000","last_became_active":"2024-03-24T11:42:23.695089+0000","last_became_peered":"2024-03-24T11:42:23.695089+0000","last_unstale":"2024-03-24T11:43:00.013815+0000","last_undegraded":"2024-03-24T11:43:00.013815+0000","last_fullsized":"2024-03-24T11:43:00.013815+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:26.847185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:33.912934+0000","last_clean_scrub_stamp":"2024-03-24T11:42:26.847185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:48.668825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":968,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404391+0000","last_change":"2024-03-24T11:42:10.116945+0000","last_active":"2024-03-24T11:42:59.404391+0000","last_peered":"2024-03-24T11:42:59.404391+0000","last_clean":"2024-03-24T11:42:59.404391+0000","last_became_active":"2024-03-24T11:41:12.184410+0000","last_became_peered":"2024-03-24T11:41:12.184410+0000","last_unstale":"2024-03-24T11:42:59.404391+0000","last_undegraded":"2024-03-24T11:42:59.404391+0000","last_fullsized":"2024-03-24T11:42:59.404391+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:10.116836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:10.116836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:21.199601+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"503'15","reported_seq":726,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013581+0000","last_change":"2024-03-24T11:42:23.700475+0000","last_active":"2024-03-24T11:43:00.013581+0000","last_peered":"2024-03-24T11:43:00.013581+0000","last_clean":"2024-03-24T11:43:00.013581+0000","last_became_active":"2024-03-24T11:42:23.699377+0000","last_became_peered":"2024-03-24T11:42:23.699377+0000","last_unstale":"2024-03-24T11:43:00.013581+0000","last_undegraded":"2024-03-24T11:43:00.013581+0000","last_fullsized":"2024-03-24T11:43:00.013581+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'14","last_scrub_stamp":"2024-03-24T11:42:07.324847+0000","last_deep_scrub":"477'14","last_deep_scrub_stamp":"2024-03-24T11:42:07.324847+0000","last_clean_scrub_stamp":"2024-03-24T11:42:07.324847+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:36.364813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"421'46","reported_seq":1110,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401509+0000","last_change":"2024-03-24T11:42:28.353945+0000","last_active":"2024-03-24T11:42:59.401509+0000","last_peered":"2024-03-24T11:42:59.401509+0000","last_clean":"2024-03-24T11:42:59.401509+0000","last_became_active":"2024-03-24T11:41:12.171159+0000","last_became_peered":"2024-03-24T11:41:12.171159+0000","last_unstale":"2024-03-24T11:42:59.401509+0000","last_undegraded":"2024-03-24T11:42:59.401509+0000","last_fullsized":"2024-03-24T11:42:59.401509+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'46","last_scrub_stamp":"2024-03-24T11:42:28.353823+0000","last_deep_scrub":"129'22","last_deep_scrub_stamp":"2024-03-24T11:23:54.165277+0000","last_clean_scrub_stamp":"2024-03-24T11:42:28.353823+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:47.808566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"145'4","reported_seq":436,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014022+0000","last_change":"2024-03-24T11:42:23.700259+0000","last_active":"2024-03-24T11:43:00.014022+0000","last_peered":"2024-03-24T11:43:00.014022+0000","last_clean":"2024-03-24T11:43:00.014022+0000","last_became_active":"2024-03-24T11:42:23.699896+0000","last_became_peered":"2024-03-24T11:42:23.699896+0000","last_unstale":"2024-03-24T11:43:00.014022+0000","last_undegraded":"2024-03-24T11:43:00.014022+0000","last_fullsized":"2024-03-24T11:43:00.014022+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'4","last_scrub_stamp":"2024-03-24T11:42:19.062303+0000","last_deep_scrub":"145'4","last_deep_scrub_stamp":"2024-03-24T11:24:46.434656+0000","last_clean_scrub_stamp":"2024-03-24T11:42:19.062303+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:26.846716+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":990,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401969+0000","last_change":"2024-03-24T11:42:33.347314+0000","last_active":"2024-03-24T11:42:59.401969+0000","last_peered":"2024-03-24T11:42:59.401969+0000","last_clean":"2024-03-24T11:42:59.401969+0000","last_became_active":"2024-03-24T11:19:00.311665+0000","last_became_peered":"2024-03-24T11:19:00.311665+0000","last_unstale":"2024-03-24T11:42:59.401969+0000","last_undegraded":"2024-03-24T11:42:59.401969+0000","last_fullsized":"2024-03-24T11:42:59.401969+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:33.347218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:33.347218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:57.407696+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"421'21","reported_seq":1022,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404759+0000","last_change":"2024-03-24T11:42:15.893463+0000","last_active":"2024-03-24T11:42:59.404759+0000","last_peered":"2024-03-24T11:42:59.404759+0000","last_clean":"2024-03-24T11:42:59.404759+0000","last_became_active":"2024-03-24T11:41:12.183577+0000","last_became_peered":"2024-03-24T11:41:12.183577+0000","last_unstale":"2024-03-24T11:42:59.404759+0000","last_undegraded":"2024-03-24T11:42:59.404759+0000","last_fullsized":"2024-03-24T11:42:59.404759+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:15.893371+0000","last_deep_scrub":"186'9","last_deep_scrub_stamp":"2024-03-24T11:27:33.522057+0000","last_clean_scrub_stamp":"2024-03-24T11:42:15.893371+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:22.725670+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1025,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976309+0000","last_change":"2024-03-24T11:42:48.340086+0000","last_active":"2024-03-24T11:42:59.976309+0000","last_peered":"2024-03-24T11:42:59.976309+0000","last_clean":"2024-03-24T11:42:59.976309+0000","last_became_active":"2024-03-24T11:18:53.682923+0000","last_became_peered":"2024-03-24T11:18:53.682923+0000","last_unstale":"2024-03-24T11:42:59.976309+0000","last_undegraded":"2024-03-24T11:42:59.976309+0000","last_fullsized":"2024-03-24T11:42:59.976309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:48.339988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:48.339988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:12.014567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":984,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402295+0000","last_change":"2024-03-24T11:42:02.172667+0000","last_active":"2024-03-24T11:42:59.402295+0000","last_peered":"2024-03-24T11:42:59.402295+0000","last_clean":"2024-03-24T11:42:59.402295+0000","last_became_active":"2024-03-24T11:18:57.693622+0000","last_became_peered":"2024-03-24T11:18:57.693622+0000","last_unstale":"2024-03-24T11:42:59.402295+0000","last_undegraded":"2024-03-24T11:42:59.402295+0000","last_fullsized":"2024-03-24T11:42:59.402295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:02.172558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:02.172558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:24.966201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":575,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013861+0000","last_change":"2024-03-24T11:42:23.702841+0000","last_active":"2024-03-24T11:43:00.013861+0000","last_peered":"2024-03-24T11:43:00.013861+0000","last_clean":"2024-03-24T11:43:00.013861+0000","last_became_active":"2024-03-24T11:42:23.702317+0000","last_became_peered":"2024-03-24T11:42:23.702317+0000","last_unstale":"2024-03-24T11:43:00.013861+0000","last_undegraded":"2024-03-24T11:43:00.013861+0000","last_fullsized":"2024-03-24T11:43:00.013861+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:57.246929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:40:24.442599+0000","last_clean_scrub_stamp":"2024-03-24T11:41:57.246929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:19.661822+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"421'25","reported_seq":1067,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014123+0000","last_change":"2024-03-24T11:42:18.451180+0000","last_active":"2024-03-24T11:43:00.014123+0000","last_peered":"2024-03-24T11:43:00.014123+0000","last_clean":"2024-03-24T11:43:00.014123+0000","last_became_active":"2024-03-24T11:41:12.175139+0000","last_became_peered":"2024-03-24T11:41:12.175139+0000","last_unstale":"2024-03-24T11:43:00.014123+0000","last_undegraded":"2024-03-24T11:43:00.014123+0000","last_fullsized":"2024-03-24T11:43:00.014123+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'25","last_scrub_stamp":"2024-03-24T11:42:18.451052+0000","last_deep_scrub":"421'25","last_deep_scrub_stamp":"2024-03-24T11:41:06.190172+0000","last_clean_scrub_stamp":"2024-03-24T11:42:18.451052+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.425395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":224,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013805+0000","last_change":"2024-03-24T11:42:23.700877+0000","last_active":"2024-03-24T11:43:00.013805+0000","last_peered":"2024-03-24T11:43:00.013805+0000","last_clean":"2024-03-24T11:43:00.013805+0000","last_became_active":"2024-03-24T11:42:23.700510+0000","last_became_peered":"2024-03-24T11:42:23.700510+0000","last_unstale":"2024-03-24T11:43:00.013805+0000","last_undegraded":"2024-03-24T11:43:00.013805+0000","last_fullsized":"2024-03-24T11:43:00.013805+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:55.105116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:55.105116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:06.529170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":972,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404744+0000","last_change":"2024-03-24T11:42:03.027873+0000","last_active":"2024-03-24T11:42:59.404744+0000","last_peered":"2024-03-24T11:42:59.404744+0000","last_clean":"2024-03-24T11:42:59.404744+0000","last_became_active":"2024-03-24T11:41:12.183024+0000","last_became_peered":"2024-03-24T11:41:12.183024+0000","last_unstale":"2024-03-24T11:42:59.404744+0000","last_undegraded":"2024-03-24T11:42:59.404744+0000","last_fullsized":"2024-03-24T11:42:59.404744+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:03.027789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:42.301950+0000","last_clean_scrub_stamp":"2024-03-24T11:42:03.027789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:23.047893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1029,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404787+0000","last_change":"2024-03-24T11:42:26.319534+0000","last_active":"2024-03-24T11:42:59.404787+0000","last_peered":"2024-03-24T11:42:59.404787+0000","last_clean":"2024-03-24T11:42:59.404787+0000","last_became_active":"2024-03-24T11:41:20.421448+0000","last_became_peered":"2024-03-24T11:41:20.421448+0000","last_unstale":"2024-03-24T11:42:59.404787+0000","last_undegraded":"2024-03-24T11:42:59.404787+0000","last_fullsized":"2024-03-24T11:42:59.404787+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:26.319444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:26.319444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:30.966523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"421'21","reported_seq":1079,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402506+0000","last_change":"2024-03-24T11:42:39.439372+0000","last_active":"2024-03-24T11:42:59.402506+0000","last_peered":"2024-03-24T11:42:59.402506+0000","last_clean":"2024-03-24T11:42:59.402506+0000","last_became_active":"2024-03-24T11:41:12.174846+0000","last_became_peered":"2024-03-24T11:41:12.174846+0000","last_unstale":"2024-03-24T11:42:59.402506+0000","last_undegraded":"2024-03-24T11:42:59.402506+0000","last_fullsized":"2024-03-24T11:42:59.402506+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:39.439182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:39.439182+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:39.780431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1013,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013098+0000","last_change":"2024-03-24T11:42:38.579733+0000","last_active":"2024-03-24T11:43:00.013098+0000","last_peered":"2024-03-24T11:43:00.013098+0000","last_clean":"2024-03-24T11:43:00.013098+0000","last_became_active":"2024-03-24T11:41:12.169201+0000","last_became_peered":"2024-03-24T11:41:12.169201+0000","last_unstale":"2024-03-24T11:43:00.013098+0000","last_undegraded":"2024-03-24T11:43:00.013098+0000","last_fullsized":"2024-03-24T11:43:00.013098+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:38.579645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:38.579645+0000","last_clean_scrub_stamp":"2024-03-24T11:42:38.579645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.350314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"469'10","reported_seq":1288,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:04.005306+0000","last_change":"2024-03-24T11:42:38.688123+0000","last_active":"2024-03-24T11:43:04.005306+0000","last_peered":"2024-03-24T11:43:04.005306+0000","last_clean":"2024-03-24T11:43:04.005306+0000","last_became_active":"2024-03-24T11:41:12.180989+0000","last_became_peered":"2024-03-24T11:41:12.180989+0000","last_unstale":"2024-03-24T11:43:04.005306+0000","last_undegraded":"2024-03-24T11:43:04.005306+0000","last_fullsized":"2024-03-24T11:43:04.005306+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'10","last_scrub_stamp":"2024-03-24T11:42:38.688025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:38.688025+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:51.896538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577531+0000","last_change":"2024-03-24T11:42:30.151531+0000","last_active":"2024-03-24T11:42:59.577531+0000","last_peered":"2024-03-24T11:42:59.577531+0000","last_clean":"2024-03-24T11:42:59.577531+0000","last_became_active":"2024-03-24T11:41:20.439380+0000","last_became_peered":"2024-03-24T11:41:20.439380+0000","last_unstale":"2024-03-24T11:42:59.577531+0000","last_undegraded":"2024-03-24T11:42:59.577531+0000","last_fullsized":"2024-03-24T11:42:59.577531+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:30.151432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:26:39.812259+0000","last_clean_scrub_stamp":"2024-03-24T11:42:30.151432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:45.535371+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"421'30","reported_seq":1045,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404345+0000","last_change":"2024-03-24T11:41:57.084015+0000","last_active":"2024-03-24T11:42:59.404345+0000","last_peered":"2024-03-24T11:42:59.404345+0000","last_clean":"2024-03-24T11:42:59.404345+0000","last_became_active":"2024-03-24T11:41:12.181695+0000","last_became_peered":"2024-03-24T11:41:12.181695+0000","last_unstale":"2024-03-24T11:42:59.404345+0000","last_undegraded":"2024-03-24T11:42:59.404345+0000","last_fullsized":"2024-03-24T11:42:59.404345+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:41:57.083911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:57.083911+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:10.767591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":232,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013691+0000","last_change":"2024-03-24T11:42:43.033863+0000","last_active":"2024-03-24T11:43:00.013691+0000","last_peered":"2024-03-24T11:43:00.013691+0000","last_clean":"2024-03-24T11:43:00.013691+0000","last_became_active":"2024-03-24T11:42:23.703737+0000","last_became_peered":"2024-03-24T11:42:23.703737+0000","last_unstale":"2024-03-24T11:43:00.013691+0000","last_undegraded":"2024-03-24T11:43:00.013691+0000","last_fullsized":"2024-03-24T11:43:00.013691+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:43.033756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:43.033756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:00.813656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"504'9","reported_seq":1293,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:04.005701+0000","last_change":"2024-03-24T11:42:27.314508+0000","last_active":"2024-03-24T11:43:04.005701+0000","last_peered":"2024-03-24T11:43:04.005701+0000","last_clean":"2024-03-24T11:43:04.005701+0000","last_became_active":"2024-03-24T11:18:57.695066+0000","last_became_peered":"2024-03-24T11:18:57.695066+0000","last_unstale":"2024-03-24T11:43:04.005701+0000","last_undegraded":"2024-03-24T11:43:04.005701+0000","last_fullsized":"2024-03-24T11:43:04.005701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'9","last_scrub_stamp":"2024-03-24T11:42:27.314365+0000","last_deep_scrub":"504'9","last_deep_scrub_stamp":"2024-03-24T11:42:27.314365+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.314365+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:51.825564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":992,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013902+0000","last_change":"2024-03-24T11:42:37.676031+0000","last_active":"2024-03-24T11:43:00.013902+0000","last_peered":"2024-03-24T11:43:00.013902+0000","last_clean":"2024-03-24T11:43:00.013902+0000","last_became_active":"2024-03-24T11:41:12.175157+0000","last_became_peered":"2024-03-24T11:41:12.175157+0000","last_unstale":"2024-03-24T11:43:00.013902+0000","last_undegraded":"2024-03-24T11:43:00.013902+0000","last_fullsized":"2024-03-24T11:43:00.013902+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:37.675933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:37.675933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:47.284048+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"421'32","reported_seq":113,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576991+0000","last_change":"2024-03-24T11:42:54.028771+0000","last_active":"2024-03-24T11:42:59.576991+0000","last_peered":"2024-03-24T11:42:59.576991+0000","last_clean":"2024-03-24T11:42:59.576991+0000","last_became_active":"2024-03-24T11:41:20.430421+0000","last_became_peered":"2024-03-24T11:41:20.430421+0000","last_unstale":"2024-03-24T11:42:59.576991+0000","last_undegraded":"2024-03-24T11:42:59.576991+0000","last_fullsized":"2024-03-24T11:42:59.576991+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:42:54.028629+0000","last_deep_scrub":"421'32","last_deep_scrub_stamp":"2024-03-24T11:42:54.028629+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.028629+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:23.508120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576940+0000","last_change":"2024-03-24T11:42:35.894966+0000","last_active":"2024-03-24T11:42:59.576940+0000","last_peered":"2024-03-24T11:42:59.576940+0000","last_clean":"2024-03-24T11:42:59.576940+0000","last_became_active":"2024-03-24T11:41:20.434013+0000","last_became_peered":"2024-03-24T11:41:20.434013+0000","last_unstale":"2024-03-24T11:42:59.576940+0000","last_undegraded":"2024-03-24T11:42:59.576940+0000","last_fullsized":"2024-03-24T11:42:59.576940+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:35.894863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:35.894863+0000","last_clean_scrub_stamp":"2024-03-24T11:42:35.894863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:49.904901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1001,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013617+0000","last_change":"2024-03-24T11:42:27.088898+0000","last_active":"2024-03-24T11:43:00.013617+0000","last_peered":"2024-03-24T11:43:00.013617+0000","last_clean":"2024-03-24T11:43:00.013617+0000","last_became_active":"2024-03-24T11:18:57.702648+0000","last_became_peered":"2024-03-24T11:18:57.702648+0000","last_unstale":"2024-03-24T11:43:00.013617+0000","last_undegraded":"2024-03-24T11:43:00.013617+0000","last_fullsized":"2024-03-24T11:43:00.013617+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:27.088809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:28.018183+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.088809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.637250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":964,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403627+0000","last_change":"2024-03-24T11:42:58.554146+0000","last_active":"2024-03-24T11:42:59.403627+0000","last_peered":"2024-03-24T11:42:59.403627+0000","last_clean":"2024-03-24T11:42:59.403627+0000","last_became_active":"2024-03-24T11:18:59.713175+0000","last_became_peered":"2024-03-24T11:18:59.713175+0000","last_unstale":"2024-03-24T11:42:59.403627+0000","last_undegraded":"2024-03-24T11:42:59.403627+0000","last_fullsized":"2024-03-24T11:42:59.403627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:58.554018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:58.554018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:21.230624+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"421'21","reported_seq":1031,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013803+0000","last_change":"2024-03-24T11:42:21.147328+0000","last_active":"2024-03-24T11:43:00.013803+0000","last_peered":"2024-03-24T11:43:00.013803+0000","last_clean":"2024-03-24T11:43:00.013803+0000","last_became_active":"2024-03-24T11:18:55.683470+0000","last_became_peered":"2024-03-24T11:18:55.683470+0000","last_unstale":"2024-03-24T11:43:00.013803+0000","last_undegraded":"2024-03-24T11:43:00.013803+0000","last_fullsized":"2024-03-24T11:43:00.013803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:42:21.147218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:21.147218+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:28.683459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":558,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:04.943398+0000","last_change":"2024-03-24T11:43:04.943424+0000","last_active":"2024-03-24T11:43:04.943398+0000","last_peered":"2024-03-24T11:43:04.943398+0000","last_clean":"2024-03-24T11:43:04.943398+0000","last_became_active":"2024-03-24T11:42:23.700882+0000","last_became_peered":"2024-03-24T11:42:23.700882+0000","last_unstale":"2024-03-24T11:43:04.943398+0000","last_undegraded":"2024-03-24T11:43:04.943398+0000","last_fullsized":"2024-03-24T11:43:04.943398+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:04.943320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:43:04.943320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:33.432873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":968,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:08.358900+0000","last_change":"2024-03-24T11:43:08.358936+0000","last_active":"2024-03-24T11:43:08.358900+0000","last_peered":"2024-03-24T11:43:08.358900+0000","last_clean":"2024-03-24T11:43:08.358900+0000","last_became_active":"2024-03-24T11:18:57.695136+0000","last_became_peered":"2024-03-24T11:18:57.695136+0000","last_unstale":"2024-03-24T11:43:08.358900+0000","last_undegraded":"2024-03-24T11:43:08.358900+0000","last_fullsized":"2024-03-24T11:43:08.358900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:08.358823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:43:08.358823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:08.643702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":989,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:03.613976+0000","last_change":"2024-03-24T11:43:03.614001+0000","last_active":"2024-03-24T11:43:03.613976+0000","last_peered":"2024-03-24T11:43:03.613976+0000","last_clean":"2024-03-24T11:43:03.613976+0000","last_became_active":"2024-03-24T11:18:59.713316+0000","last_became_peered":"2024-03-24T11:18:59.713316+0000","last_unstale":"2024-03-24T11:43:03.613976+0000","last_undegraded":"2024-03-24T11:43:03.613976+0000","last_fullsized":"2024-03-24T11:43:03.613976+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:03.613874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:03.613874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:26.395898+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"421'27","reported_seq":1043,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405019+0000","last_change":"2024-03-24T11:41:58.093979+0000","last_active":"2024-03-24T11:42:59.405019+0000","last_peered":"2024-03-24T11:42:59.405019+0000","last_clean":"2024-03-24T11:42:59.405019+0000","last_became_active":"2024-03-24T11:41:12.181398+0000","last_became_peered":"2024-03-24T11:41:12.181398+0000","last_unstale":"2024-03-24T11:42:59.405019+0000","last_undegraded":"2024-03-24T11:42:59.405019+0000","last_fullsized":"2024-03-24T11:42:59.405019+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'27","last_scrub_stamp":"2024-03-24T11:41:58.093860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:58.093860+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:23.139381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":921,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404560+0000","last_change":"2024-03-24T11:42:48.418666+0000","last_active":"2024-03-24T11:42:59.404560+0000","last_peered":"2024-03-24T11:42:59.404560+0000","last_clean":"2024-03-24T11:42:59.404560+0000","last_became_active":"2024-03-24T11:41:48.525793+0000","last_became_peered":"2024-03-24T11:41:48.525793+0000","last_unstale":"2024-03-24T11:42:59.404560+0000","last_undegraded":"2024-03-24T11:42:59.404560+0000","last_fullsized":"2024-03-24T11:42:59.404560+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:48.418020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:48.418020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:07.313849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":232,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013338+0000","last_change":"2024-03-24T11:42:36.967791+0000","last_active":"2024-03-24T11:43:00.013338+0000","last_peered":"2024-03-24T11:43:00.013338+0000","last_clean":"2024-03-24T11:43:00.013338+0000","last_became_active":"2024-03-24T11:42:23.697581+0000","last_became_peered":"2024-03-24T11:42:23.697581+0000","last_unstale":"2024-03-24T11:43:00.013338+0000","last_undegraded":"2024-03-24T11:43:00.013338+0000","last_fullsized":"2024-03-24T11:43:00.013338+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:36.967679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:39:06.116941+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.967679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:04.305581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":964,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404298+0000","last_change":"2024-03-24T11:42:14.989707+0000","last_active":"2024-03-24T11:42:59.404298+0000","last_peered":"2024-03-24T11:42:59.404298+0000","last_clean":"2024-03-24T11:42:59.404298+0000","last_became_active":"2024-03-24T11:19:00.314029+0000","last_became_peered":"2024-03-24T11:19:00.314029+0000","last_unstale":"2024-03-24T11:42:59.404298+0000","last_undegraded":"2024-03-24T11:42:59.404298+0000","last_fullsized":"2024-03-24T11:42:59.404298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:14.989611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:14.989611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:31.105756+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"421'14","reported_seq":984,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403300+0000","last_change":"2024-03-24T11:41:52.126576+0000","last_active":"2024-03-24T11:42:59.403300+0000","last_peered":"2024-03-24T11:42:59.403300+0000","last_clean":"2024-03-24T11:42:59.403300+0000","last_became_active":"2024-03-24T11:18:55.681668+0000","last_became_peered":"2024-03-24T11:18:55.681668+0000","last_unstale":"2024-03-24T11:42:59.403300+0000","last_undegraded":"2024-03-24T11:42:59.403300+0000","last_fullsized":"2024-03-24T11:42:59.403300+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'14","last_scrub_stamp":"2024-03-24T11:41:52.126490+0000","last_deep_scrub":"87'6","last_deep_scrub_stamp":"2024-03-24T11:20:14.327154+0000","last_clean_scrub_stamp":"2024-03-24T11:41:52.126490+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:04.530316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1001,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013577+0000","last_change":"2024-03-24T11:42:35.601012+0000","last_active":"2024-03-24T11:43:00.013577+0000","last_peered":"2024-03-24T11:43:00.013577+0000","last_clean":"2024-03-24T11:43:00.013577+0000","last_became_active":"2024-03-24T11:18:53.682289+0000","last_became_peered":"2024-03-24T11:18:53.682289+0000","last_unstale":"2024-03-24T11:43:00.013577+0000","last_undegraded":"2024-03-24T11:43:00.013577+0000","last_fullsized":"2024-03-24T11:43:00.013577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:35.600871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:35.600871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.131629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1033,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.976607+0000","last_change":"2024-03-24T11:42:04.285879+0000","last_active":"2024-03-24T11:42:59.976607+0000","last_peered":"2024-03-24T11:42:59.976607+0000","last_clean":"2024-03-24T11:42:59.976607+0000","last_became_active":"2024-03-24T11:41:20.433021+0000","last_became_peered":"2024-03-24T11:41:20.433021+0000","last_unstale":"2024-03-24T11:42:59.976607+0000","last_undegraded":"2024-03-24T11:42:59.976607+0000","last_fullsized":"2024-03-24T11:42:59.976607+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:04.285779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:36:56.727031+0000","last_clean_scrub_stamp":"2024-03-24T11:42:04.285779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:18.062036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":347,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.867222+0000","last_change":"2024-03-24T11:43:00.867247+0000","last_active":"2024-03-24T11:43:00.867222+0000","last_peered":"2024-03-24T11:43:00.867222+0000","last_clean":"2024-03-24T11:43:00.867222+0000","last_became_active":"2024-03-24T11:42:23.701966+0000","last_became_peered":"2024-03-24T11:42:23.701966+0000","last_unstale":"2024-03-24T11:43:00.867222+0000","last_undegraded":"2024-03-24T11:43:00.867222+0000","last_fullsized":"2024-03-24T11:43:00.867222+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:00.867112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:00.867112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:03.418480+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"421'32","reported_seq":1062,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977214+0000","last_change":"2024-03-24T11:42:23.378682+0000","last_active":"2024-03-24T11:42:59.977214+0000","last_peered":"2024-03-24T11:42:59.977214+0000","last_clean":"2024-03-24T11:42:59.977214+0000","last_became_active":"2024-03-24T11:18:55.686432+0000","last_became_peered":"2024-03-24T11:18:55.686432+0000","last_unstale":"2024-03-24T11:42:59.977214+0000","last_undegraded":"2024-03-24T11:42:59.977214+0000","last_fullsized":"2024-03-24T11:42:59.977214+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'32","last_scrub_stamp":"2024-03-24T11:42:23.378553+0000","last_deep_scrub":"421'32","last_deep_scrub_stamp":"2024-03-24T11:42:23.378553+0000","last_clean_scrub_stamp":"2024-03-24T11:42:23.378553+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:30.934465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":968,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403345+0000","last_change":"2024-03-24T11:41:51.145690+0000","last_active":"2024-03-24T11:42:59.403345+0000","last_peered":"2024-03-24T11:42:59.403345+0000","last_clean":"2024-03-24T11:42:59.403345+0000","last_became_active":"2024-03-24T11:18:53.681012+0000","last_became_peered":"2024-03-24T11:18:53.681012+0000","last_unstale":"2024-03-24T11:42:59.403345+0000","last_undegraded":"2024-03-24T11:42:59.403345+0000","last_fullsized":"2024-03-24T11:42:59.403345+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:51.145592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:41:51.145592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1007,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014170+0000","last_change":"2024-03-24T11:42:50.675604+0000","last_active":"2024-03-24T11:43:00.014170+0000","last_peered":"2024-03-24T11:43:00.014170+0000","last_clean":"2024-03-24T11:43:00.014170+0000","last_became_active":"2024-03-24T11:41:12.174243+0000","last_became_peered":"2024-03-24T11:41:12.174243+0000","last_unstale":"2024-03-24T11:43:00.014170+0000","last_undegraded":"2024-03-24T11:43:00.014170+0000","last_fullsized":"2024-03-24T11:43:00.014170+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:50.675523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:25.234374+0000","last_clean_scrub_stamp":"2024-03-24T11:42:50.675523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:09.952710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1008,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:02.517662+0000","last_change":"2024-03-24T11:43:02.517686+0000","last_active":"2024-03-24T11:43:02.517662+0000","last_peered":"2024-03-24T11:43:02.517662+0000","last_clean":"2024-03-24T11:43:02.517662+0000","last_became_active":"2024-03-24T11:41:20.434440+0000","last_became_peered":"2024-03-24T11:41:20.434440+0000","last_unstale":"2024-03-24T11:43:02.517662+0000","last_undegraded":"2024-03-24T11:43:02.517662+0000","last_fullsized":"2024-03-24T11:43:02.517662+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:02.517584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:40.280347+0000","last_clean_scrub_stamp":"2024-03-24T11:43:02.517584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:30.427027+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"421'30","reported_seq":1106,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014022+0000","last_change":"2024-03-24T11:42:54.726626+0000","last_active":"2024-03-24T11:43:00.014022+0000","last_peered":"2024-03-24T11:43:00.014022+0000","last_clean":"2024-03-24T11:43:00.014022+0000","last_became_active":"2024-03-24T11:41:12.176633+0000","last_became_peered":"2024-03-24T11:41:12.176633+0000","last_unstale":"2024-03-24T11:43:00.014022+0000","last_undegraded":"2024-03-24T11:43:00.014022+0000","last_fullsized":"2024-03-24T11:43:00.014022+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'30","last_scrub_stamp":"2024-03-24T11:42:54.726527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:54.726527+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.033482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":1029,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012753+0000","last_change":"2024-03-24T11:42:04.123992+0000","last_active":"2024-03-24T11:43:00.012753+0000","last_peered":"2024-03-24T11:43:00.012753+0000","last_clean":"2024-03-24T11:43:00.012753+0000","last_became_active":"2024-03-24T11:41:20.436875+0000","last_became_peered":"2024-03-24T11:41:20.436875+0000","last_unstale":"2024-03-24T11:43:00.012753+0000","last_undegraded":"2024-03-24T11:43:00.012753+0000","last_fullsized":"2024-03-24T11:43:00.012753+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-24T11:42:04.123899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:04.123899+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:19.688388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"477'13","reported_seq":1356,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.977158+0000","last_change":"2024-03-24T11:42:19.445563+0000","last_active":"2024-03-24T11:42:59.977158+0000","last_peered":"2024-03-24T11:42:59.977158+0000","last_clean":"2024-03-24T11:42:59.977158+0000","last_became_active":"2024-03-24T11:41:20.431908+0000","last_became_peered":"2024-03-24T11:41:20.431908+0000","last_unstale":"2024-03-24T11:42:59.977158+0000","last_undegraded":"2024-03-24T11:42:59.977158+0000","last_fullsized":"2024-03-24T11:42:59.977158+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'13","last_scrub_stamp":"2024-03-24T11:42:19.445321+0000","last_deep_scrub":"87'9","last_deep_scrub_stamp":"2024-03-24T11:35:56.346906+0000","last_clean_scrub_stamp":"2024-03-24T11:42:19.445321+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:49.130407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"88'2","reported_seq":584,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013631+0000","last_change":"2024-03-24T11:42:23.704163+0000","last_active":"2024-03-24T11:43:00.013631+0000","last_peered":"2024-03-24T11:43:00.013631+0000","last_clean":"2024-03-24T11:43:00.013631+0000","last_became_active":"2024-03-24T11:42:23.703678+0000","last_became_peered":"2024-03-24T11:42:23.703678+0000","last_unstale":"2024-03-24T11:43:00.013631+0000","last_undegraded":"2024-03-24T11:43:00.013631+0000","last_fullsized":"2024-03-24T11:43:00.013631+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-24T11:41:53.139987+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-24T11:41:53.139987+0000","last_clean_scrub_stamp":"2024-03-24T11:41:53.139987+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:09.895781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"421'24","reported_seq":1032,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403816+0000","last_change":"2024-03-24T11:41:32.341653+0000","last_active":"2024-03-24T11:42:59.403816+0000","last_peered":"2024-03-24T11:42:59.403816+0000","last_clean":"2024-03-24T11:42:59.403816+0000","last_became_active":"2024-03-24T11:41:20.436384+0000","last_became_peered":"2024-03-24T11:41:20.436384+0000","last_unstale":"2024-03-24T11:42:59.403816+0000","last_undegraded":"2024-03-24T11:42:59.403816+0000","last_fullsized":"2024-03-24T11:42:59.403816+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'24","last_scrub_stamp":"2024-03-24T11:41:32.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:32.341564+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576715+0000","last_change":"2024-03-24T11:42:27.987032+0000","last_active":"2024-03-24T11:42:59.576715+0000","last_peered":"2024-03-24T11:42:59.576715+0000","last_clean":"2024-03-24T11:42:59.576715+0000","last_became_active":"2024-03-24T11:41:20.427341+0000","last_became_peered":"2024-03-24T11:41:20.427341+0000","last_unstale":"2024-03-24T11:42:59.576715+0000","last_undegraded":"2024-03-24T11:42:59.576715+0000","last_fullsized":"2024-03-24T11:42:59.576715+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:27.986921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:11.852848+0000","last_clean_scrub_stamp":"2024-03-24T11:42:27.986921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:43.072514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576943+0000","last_change":"2024-03-24T11:42:55.054792+0000","last_active":"2024-03-24T11:42:59.576943+0000","last_peered":"2024-03-24T11:42:59.576943+0000","last_clean":"2024-03-24T11:42:59.576943+0000","last_became_active":"2024-03-24T11:41:20.439681+0000","last_became_peered":"2024-03-24T11:41:20.439681+0000","last_unstale":"2024-03-24T11:42:59.576943+0000","last_undegraded":"2024-03-24T11:42:59.576943+0000","last_fullsized":"2024-03-24T11:42:59.576943+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:55.054677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:55.054677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:15.892918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":960,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404470+0000","last_change":"2024-03-24T11:42:12.097323+0000","last_active":"2024-03-24T11:42:59.404470+0000","last_peered":"2024-03-24T11:42:59.404470+0000","last_clean":"2024-03-24T11:42:59.404470+0000","last_became_active":"2024-03-24T11:19:00.311808+0000","last_became_peered":"2024-03-24T11:19:00.311808+0000","last_unstale":"2024-03-24T11:42:59.404470+0000","last_undegraded":"2024-03-24T11:42:59.404470+0000","last_fullsized":"2024-03-24T11:42:59.404470+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:12.097225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:12.097225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:28.933272+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"421'39","reported_seq":1081,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404477+0000","last_change":"2024-03-24T11:42:34.388539+0000","last_active":"2024-03-24T11:42:59.404477+0000","last_peered":"2024-03-24T11:42:59.404477+0000","last_clean":"2024-03-24T11:42:59.404477+0000","last_became_active":"2024-03-24T11:18:55.684511+0000","last_became_peered":"2024-03-24T11:18:55.684511+0000","last_unstale":"2024-03-24T11:42:59.404477+0000","last_undegraded":"2024-03-24T11:42:59.404477+0000","last_fullsized":"2024-03-24T11:42:59.404477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:42:34.388433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:34.388433+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:58.868559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1024,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013107+0000","last_change":"2024-03-24T11:42:47.634770+0000","last_active":"2024-03-24T11:43:00.013107+0000","last_peered":"2024-03-24T11:43:00.013107+0000","last_clean":"2024-03-24T11:43:00.013107+0000","last_became_active":"2024-03-24T11:41:20.431934+0000","last_became_peered":"2024-03-24T11:41:20.431934+0000","last_unstale":"2024-03-24T11:43:00.013107+0000","last_undegraded":"2024-03-24T11:43:00.013107+0000","last_fullsized":"2024-03-24T11:43:00.013107+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:47.634682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:42:47.634682+0000","last_clean_scrub_stamp":"2024-03-24T11:42:47.634682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:55.277468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1004,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.014069+0000","last_change":"2024-03-24T11:42:49.144640+0000","last_active":"2024-03-24T11:43:00.014069+0000","last_peered":"2024-03-24T11:43:00.014069+0000","last_clean":"2024-03-24T11:43:00.014069+0000","last_became_active":"2024-03-24T11:41:20.439167+0000","last_became_peered":"2024-03-24T11:41:20.439167+0000","last_unstale":"2024-03-24T11:43:00.014069+0000","last_undegraded":"2024-03-24T11:43:00.014069+0000","last_fullsized":"2024-03-24T11:43:00.014069+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:49.144471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:22:00.847219+0000","last_clean_scrub_stamp":"2024-03-24T11:42:49.144471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:54.634464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577238+0000","last_change":"2024-03-24T11:41:59.110867+0000","last_active":"2024-03-24T11:42:59.577238+0000","last_peered":"2024-03-24T11:42:59.577238+0000","last_clean":"2024-03-24T11:42:59.577238+0000","last_became_active":"2024-03-24T11:41:20.436845+0000","last_became_peered":"2024-03-24T11:41:20.436845+0000","last_unstale":"2024-03-24T11:42:59.577238+0000","last_undegraded":"2024-03-24T11:42:59.577238+0000","last_fullsized":"2024-03-24T11:42:59.577238+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:59.110715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:28:31.885946+0000","last_clean_scrub_stamp":"2024-03-24T11:41:59.110715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:23.548564+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"421'9","reported_seq":99,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.577178+0000","last_change":"2024-03-24T11:42:20.031601+0000","last_active":"2024-03-24T11:42:59.577178+0000","last_peered":"2024-03-24T11:42:59.577178+0000","last_clean":"2024-03-24T11:42:59.577178+0000","last_became_active":"2024-03-24T11:41:20.439702+0000","last_became_peered":"2024-03-24T11:41:20.439702+0000","last_unstale":"2024-03-24T11:42:59.577178+0000","last_undegraded":"2024-03-24T11:42:59.577178+0000","last_fullsized":"2024-03-24T11:42:59.577178+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'9","last_scrub_stamp":"2024-03-24T11:42:20.031489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:20.031489+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:41.536673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1004,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402175+0000","last_change":"2024-03-24T11:41:59.087942+0000","last_active":"2024-03-24T11:42:59.402175+0000","last_peered":"2024-03-24T11:42:59.402175+0000","last_clean":"2024-03-24T11:42:59.402175+0000","last_became_active":"2024-03-24T11:41:20.423037+0000","last_became_peered":"2024-03-24T11:41:20.423037+0000","last_unstale":"2024-03-24T11:42:59.402175+0000","last_undegraded":"2024-03-24T11:42:59.402175+0000","last_fullsized":"2024-03-24T11:42:59.402175+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:59.087816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:27.463218+0000","last_clean_scrub_stamp":"2024-03-24T11:41:59.087816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:24.054613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":224,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013465+0000","last_change":"2024-03-24T11:42:23.693015+0000","last_active":"2024-03-24T11:43:00.013465+0000","last_peered":"2024-03-24T11:43:00.013465+0000","last_clean":"2024-03-24T11:43:00.013465+0000","last_became_active":"2024-03-24T11:42:23.692534+0000","last_became_peered":"2024-03-24T11:42:23.692534+0000","last_unstale":"2024-03-24T11:43:00.013465+0000","last_undegraded":"2024-03-24T11:43:00.013465+0000","last_fullsized":"2024-03-24T11:43:00.013465+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:52.162832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:30.983558+0000","last_clean_scrub_stamp":"2024-03-24T11:41:52.162832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:01.616558+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":981,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401812+0000","last_change":"2024-03-24T11:42:19.322923+0000","last_active":"2024-03-24T11:42:59.401812+0000","last_peered":"2024-03-24T11:42:59.401812+0000","last_clean":"2024-03-24T11:42:59.401812+0000","last_became_active":"2024-03-24T11:18:59.706795+0000","last_became_peered":"2024-03-24T11:18:59.706795+0000","last_unstale":"2024-03-24T11:42:59.401812+0000","last_undegraded":"2024-03-24T11:42:59.401812+0000","last_fullsized":"2024-03-24T11:42:59.401812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:19.322803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:33:28.607919+0000","last_clean_scrub_stamp":"2024-03-24T11:42:19.322803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:26.121082+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"421'21","reported_seq":103,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576849+0000","last_change":"2024-03-24T11:41:46.167562+0000","last_active":"2024-03-24T11:42:59.576849+0000","last_peered":"2024-03-24T11:42:59.576849+0000","last_clean":"2024-03-24T11:42:59.576849+0000","last_became_active":"2024-03-24T11:41:20.442581+0000","last_became_peered":"2024-03-24T11:41:20.442581+0000","last_unstale":"2024-03-24T11:42:59.576849+0000","last_undegraded":"2024-03-24T11:42:59.576849+0000","last_fullsized":"2024-03-24T11:42:59.576849+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:41:46.167454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:41:46.167454+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:13.114702+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1012,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.402237+0000","last_change":"2024-03-24T11:42:46.322506+0000","last_active":"2024-03-24T11:42:59.402237+0000","last_peered":"2024-03-24T11:42:59.402237+0000","last_clean":"2024-03-24T11:42:59.402237+0000","last_became_active":"2024-03-24T11:41:20.432122+0000","last_became_peered":"2024-03-24T11:41:20.432122+0000","last_unstale":"2024-03-24T11:42:59.402237+0000","last_undegraded":"2024-03-24T11:42:59.402237+0000","last_fullsized":"2024-03-24T11:42:59.402237+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:46.322399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:25:46.067824+0000","last_clean_scrub_stamp":"2024-03-24T11:42:46.322399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:46.598188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1010,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.401870+0000","last_change":"2024-03-24T11:42:04.185684+0000","last_active":"2024-03-24T11:42:59.401870+0000","last_peered":"2024-03-24T11:42:59.401870+0000","last_clean":"2024-03-24T11:42:59.401870+0000","last_became_active":"2024-03-24T11:41:12.168113+0000","last_became_peered":"2024-03-24T11:41:12.168113+0000","last_unstale":"2024-03-24T11:42:59.401870+0000","last_undegraded":"2024-03-24T11:42:59.401870+0000","last_fullsized":"2024-03-24T11:42:59.401870+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:04.185585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:27:32.894112+0000","last_clean_scrub_stamp":"2024-03-24T11:42:04.185585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:22.138511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":992,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013031+0000","last_change":"2024-03-24T11:41:46.401947+0000","last_active":"2024-03-24T11:43:00.013031+0000","last_peered":"2024-03-24T11:43:00.013031+0000","last_clean":"2024-03-24T11:43:00.013031+0000","last_became_active":"2024-03-24T11:41:12.172311+0000","last_became_peered":"2024-03-24T11:41:12.172311+0000","last_unstale":"2024-03-24T11:43:00.013031+0000","last_undegraded":"2024-03-24T11:43:00.013031+0000","last_fullsized":"2024-03-24T11:43:00.013031+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:46.401832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:41:46.401832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:09.185024+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"421'39","reported_seq":1025,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.405072+0000","last_change":"2024-03-24T11:42:44.623567+0000","last_active":"2024-03-24T11:42:59.405072+0000","last_peered":"2024-03-24T11:42:59.405072+0000","last_clean":"2024-03-24T11:42:59.405072+0000","last_became_active":"2024-03-24T11:18:55.680611+0000","last_became_peered":"2024-03-24T11:18:55.680611+0000","last_unstale":"2024-03-24T11:42:59.405072+0000","last_undegraded":"2024-03-24T11:42:59.405072+0000","last_fullsized":"2024-03-24T11:42:59.405072+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'39","last_scrub_stamp":"2024-03-24T11:42:44.623454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:42:44.623454+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:55.873880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":970,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.403597+0000","last_change":"2024-03-24T11:42:36.656857+0000","last_active":"2024-03-24T11:42:59.403597+0000","last_peered":"2024-03-24T11:42:59.403597+0000","last_clean":"2024-03-24T11:42:59.403597+0000","last_became_active":"2024-03-24T11:18:53.679966+0000","last_became_peered":"2024-03-24T11:18:53.679966+0000","last_unstale":"2024-03-24T11:42:59.403597+0000","last_undegraded":"2024-03-24T11:42:59.403597+0000","last_fullsized":"2024-03-24T11:42:59.403597+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:36.656695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:52.658804+0000","last_clean_scrub_stamp":"2024-03-24T11:42:36.656695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:01.226380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":93,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.576442+0000","last_change":"2024-03-24T11:42:39.885660+0000","last_active":"2024-03-24T11:42:59.576442+0000","last_peered":"2024-03-24T11:42:59.576442+0000","last_clean":"2024-03-24T11:42:59.576442+0000","last_became_active":"2024-03-24T11:41:20.429824+0000","last_became_peered":"2024-03-24T11:41:20.429824+0000","last_unstale":"2024-03-24T11:42:59.576442+0000","last_undegraded":"2024-03-24T11:42:59.576442+0000","last_fullsized":"2024-03-24T11:42:59.576442+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:39.885558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:39.885558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:07.460919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":967,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:42:59.404179+0000","last_change":"2024-03-24T11:42:23.043823+0000","last_active":"2024-03-24T11:42:59.404179+0000","last_peered":"2024-03-24T11:42:59.404179+0000","last_clean":"2024-03-24T11:42:59.404179+0000","last_became_active":"2024-03-24T11:41:12.185025+0000","last_became_peered":"2024-03-24T11:41:12.185025+0000","last_unstale":"2024-03-24T11:42:59.404179+0000","last_undegraded":"2024-03-24T11:42:59.404179+0000","last_fullsized":"2024-03-24T11:42:59.404179+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:23.043726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:42:23.043726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:34.641398+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"421'21","reported_seq":111,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:07.927256+0000","last_change":"2024-03-24T11:43:07.927295+0000","last_active":"2024-03-24T11:43:07.927256+0000","last_peered":"2024-03-24T11:43:07.927256+0000","last_clean":"2024-03-24T11:43:07.927256+0000","last_became_active":"2024-03-24T11:41:20.431239+0000","last_became_peered":"2024-03-24T11:41:20.431239+0000","last_unstale":"2024-03-24T11:43:07.927256+0000","last_undegraded":"2024-03-24T11:43:07.927256+0000","last_fullsized":"2024-03-24T11:43:07.927256+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'21","last_scrub_stamp":"2024-03-24T11:43:07.927182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:54.665517+0000","last_clean_scrub_stamp":"2024-03-24T11:43:07.927182+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:14.786427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":994,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.013733+0000","last_change":"2024-03-24T11:41:50.398182+0000","last_active":"2024-03-24T11:43:00.013733+0000","last_peered":"2024-03-24T11:43:00.013733+0000","last_clean":"2024-03-24T11:43:00.013733+0000","last_became_active":"2024-03-24T11:18:53.684530+0000","last_became_peered":"2024-03-24T11:18:53.684530+0000","last_unstale":"2024-03-24T11:43:00.013733+0000","last_undegraded":"2024-03-24T11:43:00.013733+0000","last_fullsized":"2024-03-24T11:43:00.013733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:41:50.398096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:41:50.398096+0000","last_clean_scrub_stamp":"2024-03-24T11:41:50.398096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:16.468009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1004,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:00.012984+0000","last_change":"2024-03-24T11:42:20.180676+0000","last_active":"2024-03-24T11:43:00.012984+0000","last_peered":"2024-03-24T11:43:00.012984+0000","last_clean":"2024-03-24T11:43:00.012984+0000","last_became_active":"2024-03-24T11:41:12.181763+0000","last_became_peered":"2024-03-24T11:41:12.181763+0000","last_unstale":"2024-03-24T11:43:00.012984+0000","last_undegraded":"2024-03-24T11:43:00.012984+0000","last_fullsized":"2024-03-24T11:43:00.012984+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:42:20.180604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:56.669278+0000","last_clean_scrub_stamp":"2024-03-24T11:42:20.180604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:32.552068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":101,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-24T11:43:01.057616+0000","last_change":"2024-03-24T11:43:01.057640+0000","last_active":"2024-03-24T11:43:01.057616+0000","last_peered":"2024-03-24T11:43:01.057616+0000","last_clean":"2024-03-24T11:43:01.057616+0000","last_became_active":"2024-03-24T11:41:20.437643+0000","last_became_peered":"2024-03-24T11:41:20.437643+0000","last_unstale":"2024-03-24T11:43:01.057616+0000","last_undegraded":"2024-03-24T11:43:01.057616+0000","last_fullsized":"2024-03-24T11:43:01.057616+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:43:01.057531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:18:58.683079+0000","last_clean_scrub_stamp":"2024-03-24T11:43:01.057531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:44:05.222308+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":675,"num_read_kb":670,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":6694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1046,"ondisk_log_size":1046,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070760,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798996,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946156,"statfs":{"total":95995035648,"available":95176863744,"internally_reserved":0,"allocated":1519616,"data_stored":1040959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":816642507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365101,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713996,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93031156,"statfs":{"total":95995035648,"available":95263903744,"internally_reserved":0,"allocated":1179648,"data_stored":713500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575175,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938928,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806224,"statfs":{"total":95995035648,"available":95033573376,"internally_reserved":0,"allocated":1220608,"data_stored":714337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738700,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840352,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92904800,"statfs":{"total":95995035648,"available":95134515200,"internally_reserved":0,"allocated":1204224,"data_stored":713144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":859306437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411650,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722580,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93022572,"statfs":{"total":95995035648,"available":95255113728,"internally_reserved":0,"allocated":1454080,"data_stored":975483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150011,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881180,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92863972,"statfs":{"total":95995035648,"available":95092707328,"internally_reserved":0,"allocated":1789952,"data_stored":1303429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":900528583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986487,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569632,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175520,"statfs":{"total":95995035648,"available":95411732480,"internally_reserved":0,"allocated":1204224,"data_stored":713707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659442,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849248,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92895904,"statfs":{"total":95995035648,"available":95125405696,"internally_reserved":0,"allocated":1794048,"data_stored":1303787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":867826117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:43:11.991 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T11:43:11.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-24T11:43:12.250 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:43:12.636 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:12 smithi179 ceph-mon[29608]: from='client.16311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:43:12.637 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:12 smithi179 ceph-mon[37907]: from='client.16311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:43:12.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:12 smithi181 ceph-mon[33998]: from='client.16311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:43:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[37907]: 4.c scrub starts 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[37907]: 4.c scrub ok 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[37907]: 5.7 scrub starts 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[37907]: 5.7 scrub ok 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[37907]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[29608]: 4.c scrub starts 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[29608]: 4.c scrub ok 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[29608]: 5.7 scrub starts 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[29608]: 5.7 scrub ok 2024-03-24T11:43:13.710 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:13 smithi179 ceph-mon[29608]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:13 smithi181 ceph-mon[33998]: 4.c scrub starts 2024-03-24T11:43:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:13 smithi181 ceph-mon[33998]: 4.c scrub ok 2024-03-24T11:43:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:13 smithi181 ceph-mon[33998]: 5.7 scrub starts 2024-03-24T11:43:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:13 smithi181 ceph-mon[33998]: 5.7 scrub ok 2024-03-24T11:43:13.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:13 smithi181 ceph-mon[33998]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T11:43:14.552 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:43:14] ENGINE Bus STOPPING 2024-03-24T11:43:14.552 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:43:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:43:14.552 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:43:14] ENGINE Bus STOPPED 2024-03-24T11:43:14.552 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:43:14] ENGINE Bus STARTING 2024-03-24T11:43:14.553 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:43:14] ENGINE Serving on http://:::9283 2024-03-24T11:43:14.553 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: [24/Mar/2024:11:43:14] ENGINE Bus STARTED 2024-03-24T11:43:14.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug there is no tcmu-runner data available 2024-03-24T11:43:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: 5.1a scrub starts 2024-03-24T11:43:14.902 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: 5.1a scrub ok 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: 3.2 deep-scrub starts 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: 3.2 deep-scrub ok 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3999319867' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: from='client.? 172.21.15.179:0/3999319867' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:43:14.903 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:14 smithi181 ceph-mon[33998]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T11:43:14.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: 5.1a scrub starts 2024-03-24T11:43:14.952 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: 5.1a scrub ok 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: 3.2 deep-scrub starts 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: 3.2 deep-scrub ok 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3999319867' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: from='client.? 172.21.15.179:0/3999319867' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[29608]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: 5.1a scrub starts 2024-03-24T11:43:14.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: 5.1a scrub ok 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: 3.2 deep-scrub starts 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: 3.2 deep-scrub ok 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3999319867' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: from='client.? 172.21.15.179:0/3999319867' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: from='mgr.14568 172.21.15.179:0/3709382070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: from='mgr.14568 ' entity='mgr.y' 2024-03-24T11:43:14.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:14 smithi179 ceph-mon[37907]: osdmap e514: 8 total, 8 up, 8 in 2024-03-24T11:43:14.979 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T11:43:14.991 INFO:tasks.cephadm:Teardown begin 2024-03-24T11:43:14.991 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:43:15.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:43:15.081 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T11:43:15.081 DEBUG:teuthology.orchestra.run.smithi179:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:43:15.114 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:43:15.134 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T11:43:15.135 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T11:43:15.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:43:15.209 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:15 smithi179 systemd[1]: Stopping Ceph mon.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:15.209 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: ::ffff:172.21.15.181 - - [24/Mar/2024:11:43:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:43:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29604]: 2024-03-24T11:43:15.455+0000 7f36b7d5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:15 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a[29604]: 2024-03-24T11:43:15.455+0000 7f36b7d5e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T11:43:15.709 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:15 smithi179 podman[208676]: 2024-03-24 11:43:15.486864892 +0000 UTC m=+0.135627216 container died d0b2dc392a46005f899fe6011f9db7030a28614a39f1d7007feecffbe68852f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307) 2024-03-24T11:43:16.092 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:15 smithi179 podman[208676]: 2024-03-24 11:43:15.848467082 +0000 UTC m=+0.497229399 container cleanup d0b2dc392a46005f899fe6011f9db7030a28614a39f1d7007feecffbe68852f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T11:43:16.092 INFO:journalctl@ceph.mon.a.smithi179.stdout:Mar 24 11:43:15 smithi179 bash[208676]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:43:16.309 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T11:43:16.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:16.366 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T11:43:16.366 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-24T11:43:16.366 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.c 2024-03-24T11:43:16.670 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:16 smithi179 systemd[1]: Stopping Ceph mon.c for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:16.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:16 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c[37903]: 2024-03-24T11:43:16.669+0000 7fb6cf283700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:16.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:16 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c[37903]: 2024-03-24T11:43:16.669+0000 7fb6cf283700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:43:16.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:16 smithi179 podman[208797]: 2024-03-24 11:43:16.854298222 +0000 UTC m=+0.306671704 container died 13ecfd493a2dea82f8d07c9821cb7eeaa69852acf9d1e5709c78d25e6554915f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:43:17.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:17 smithi179 podman[208797]: 2024-03-24 11:43:17.346147197 +0000 UTC m=+0.798520715 container cleanup 13ecfd493a2dea82f8d07c9821cb7eeaa69852acf9d1e5709c78d25e6554915f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-24T11:43:17.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 24 11:43:17 smithi179 bash[208797]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-c 2024-03-24T11:43:17.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T11:43:17.832 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:17.832 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-24T11:43:17.832 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-24T11:43:17.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.b 2024-03-24T11:43:18.402 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:18 smithi181 systemd[1]: Stopping Ceph mon.b for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:18.741 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:18 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-b[33988]: 2024-03-24T11:43:18.413+0000 7f3d33e4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:18.741 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:18 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-b[33988]: 2024-03-24T11:43:18.413+0000 7f3d33e4c700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:43:18.741 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:18 smithi181 podman[154757]: 2024-03-24 11:43:18.505953104 +0000 UTC m=+0.197177291 container died 3ae1c3d129c5e747d3a464afdeeaa5e29179ed35edbc9d4c3cf1390e356938df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True) 2024-03-24T11:43:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:18 smithi181 podman[154757]: 2024-03-24 11:43:18.889541697 +0000 UTC m=+0.580765936 container cleanup 3ae1c3d129c5e747d3a464afdeeaa5e29179ed35edbc9d4c3cf1390e356938df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T11:43:19.152 INFO:journalctl@ceph.mon.b.smithi181.stdout:Mar 24 11:43:18 smithi181 bash[154757]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mon-b 2024-03-24T11:43:19.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T11:43:19.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:19.444 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-24T11:43:19.444 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T11:43:19.444 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:43:19.768 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:19 smithi179 systemd[1]: Stopping Ceph mgr.y for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:20.086 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:19 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y[29844]: 2024-03-24T11:43:19.767+0000 7f6812f0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:20.086 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:19 smithi179 podman[208919]: 2024-03-24 11:43:19.799613088 +0000 UTC m=+0.164996525 container died e99c22ba74426d013c47c1e790fab6ebfdbbf3b82ed1abdf44bb693e3f4888ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:43:20.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:20 smithi179 podman[208919]: 2024-03-24 11:43:20.26095634 +0000 UTC m=+0.626339774 container cleanup e99c22ba74426d013c47c1e790fab6ebfdbbf3b82ed1abdf44bb693e3f4888ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:43:20.459 INFO:journalctl@ceph.mgr.y.smithi179.stdout:Mar 24 11:43:20 smithi179 bash[208919]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-y 2024-03-24T11:43:20.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T11:43:20.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:20.733 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T11:43:20.734 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T11:43:20.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.x 2024-03-24T11:43:21.059 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:43:20 smithi181 systemd[1]: Stopping Ceph mgr.x for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:21.330 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:43:21 smithi181 podman[154880]: 2024-03-24 11:43:21.058937352 +0000 UTC m=+0.130951005 container died caff3ba4924e1878aeba3718dd79c187d94554c91cb12137c3cc41508bb76aa9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T11:43:21.652 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:43:21 smithi181 podman[154880]: 2024-03-24 11:43:21.419526627 +0000 UTC m=+0.491540291 container cleanup caff3ba4924e1878aeba3718dd79c187d94554c91cb12137c3cc41508bb76aa9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T11:43:21.652 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Mar 24 11:43:21 smithi181 bash[154880]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-mgr-x 2024-03-24T11:43:21.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T11:43:21.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:21.849 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T11:43:21.849 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T11:43:21.850 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.0 2024-03-24T11:43:22.141 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:21 smithi179 systemd[1]: Stopping Ceph osd.0 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:22.459 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0[41905]: 2024-03-24T11:43:22.140+0000 7f88a341a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:22.459 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0[41905]: 2024-03-24T11:43:22.140+0000 7f88a341a700 -1 osd.0 514 *** Got signal Terminated *** 2024-03-24T11:43:22.459 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:22 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0[41905]: 2024-03-24T11:43:22.140+0000 7f88a341a700 -1 osd.0 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:43:28.001 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:27 smithi179 podman[209039]: 2024-03-24 11:43:27.715990485 +0000 UTC m=+5.677062815 container died 71405bd0bad79473836a33b8274490827ea53abdc16b4c17ee9b9a1dc03a0e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:43:28.459 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:28 smithi179 podman[209039]: 2024-03-24 11:43:28.200131223 +0000 UTC m=+6.161203560 container cleanup 71405bd0bad79473836a33b8274490827ea53abdc16b4c17ee9b9a1dc03a0e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T11:43:28.459 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:28 smithi179 bash[209039]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:43:28.781 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:28 smithi179 podman[209070]: 2024-03-24 11:43:28.486174519 +0000 UTC m=+0.759464509 container remove 71405bd0bad79473836a33b8274490827ea53abdc16b4c17ee9b9a1dc03a0e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, ceph=True) 2024-03-24T11:43:29.149 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:28 smithi179 podman[209163]: 2024-03-24 11:43:28.685163453 +0000 UTC m=+0.018377019 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:43:29.149 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:28 smithi179 podman[209163]: 2024-03-24 11:43:28.795499842 +0000 UTC m=+0.128713388 container create cae99caa9a25947fc0d739b745dfd0f0d9c6dc45bdaad1f2a87c78fc15494029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T11:43:29.149 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:29 smithi179 podman[209163]: 2024-03-24 11:43:29.015026286 +0000 UTC m=+0.348239845 container init cae99caa9a25947fc0d739b745dfd0f0d9c6dc45bdaad1f2a87c78fc15494029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b) 2024-03-24T11:43:29.149 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:29 smithi179 podman[209163]: 2024-03-24 11:43:29.018041666 +0000 UTC m=+0.351255214 container start cae99caa9a25947fc0d739b745dfd0f0d9c6dc45bdaad1f2a87c78fc15494029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T11:43:29.149 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:29 smithi179 podman[209163]: 2024-03-24 11:43:29.148665642 +0000 UTC m=+0.481879202 container attach cae99caa9a25947fc0d739b745dfd0f0d9c6dc45bdaad1f2a87c78fc15494029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:43:29.861 INFO:journalctl@ceph.osd.0.smithi179.stdout:Mar 24 11:43:29 smithi179 podman[209163]: 2024-03-24 11:43:29.569972198 +0000 UTC m=+0.903185747 container died cae99caa9a25947fc0d739b745dfd0f0d9c6dc45bdaad1f2a87c78fc15494029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T11:43:30.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T11:43:30.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:30.365 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T11:43:30.365 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T11:43:30.365 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.1 2024-03-24T11:43:30.709 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:30 smithi179 systemd[1]: Stopping Ceph osd.1 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:31.209 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:30 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1[47887]: 2024-03-24T11:43:30.748+0000 7fed6171a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:31.209 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:30 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1[47887]: 2024-03-24T11:43:30.748+0000 7fed6171a700 -1 osd.1 514 *** Got signal Terminated *** 2024-03-24T11:43:31.210 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:30 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1[47887]: 2024-03-24T11:43:30.748+0000 7fed6171a700 -1 osd.1 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:43:36.592 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:36 smithi179 podman[209272]: 2024-03-24 11:43:36.299853125 +0000 UTC m=+5.709021622 container died cec915c18f1fdb3dc8501977639382a961ef800d8a11f14f6b7ac194c3141e3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T11:43:36.959 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:36 smithi179 podman[209272]: 2024-03-24 11:43:36.766488247 +0000 UTC m=+6.175656783 container cleanup cec915c18f1fdb3dc8501977639382a961ef800d8a11f14f6b7ac194c3141e3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:43:36.959 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:36 smithi179 bash[209272]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:43:37.373 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:37 smithi179 podman[209290]: 2024-03-24 11:43:37.068114126 +0000 UTC m=+0.767577967 container remove cec915c18f1fdb3dc8501977639382a961ef800d8a11f14f6b7ac194c3141e3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T11:43:37.638 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:37 smithi179 podman[209381]: 2024-03-24 11:43:37.27719143 +0000 UTC m=+0.018635292 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:43:37.639 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:37 smithi179 podman[209381]: 2024-03-24 11:43:37.385756574 +0000 UTC m=+0.127200431 container create df513a75ff772c0aca1ed245e24ed6859148e9599e8387b7834a85b542d62e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:43:37.639 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:37 smithi179 podman[209381]: 2024-03-24 11:43:37.555010172 +0000 UTC m=+0.296454030 container init df513a75ff772c0aca1ed245e24ed6859148e9599e8387b7834a85b542d62e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:43:37.639 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:37 smithi179 podman[209381]: 2024-03-24 11:43:37.558025207 +0000 UTC m=+0.299469067 container start df513a75ff772c0aca1ed245e24ed6859148e9599e8387b7834a85b542d62e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T11:43:37.639 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:37 smithi179 podman[209381]: 2024-03-24 11:43:37.638311113 +0000 UTC m=+0.379754971 container attach df513a75ff772c0aca1ed245e24ed6859148e9599e8387b7834a85b542d62e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T11:43:37.959 INFO:journalctl@ceph.osd.1.smithi179.stdout:Mar 24 11:43:37 smithi179 podman[209381]: 2024-03-24 11:43:37.846367607 +0000 UTC m=+0.587811467 container died df513a75ff772c0aca1ed245e24ed6859148e9599e8387b7834a85b542d62e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:43:38.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T11:43:38.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:38.588 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T11:43:38.588 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T11:43:38.588 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.2 2024-03-24T11:43:38.924 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:38 smithi179 systemd[1]: Stopping Ceph osd.2 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:39.209 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:38 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2[53906]: 2024-03-24T11:43:38.923+0000 7fd5f7d43700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:39.209 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:38 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2[53906]: 2024-03-24T11:43:38.923+0000 7fd5f7d43700 -1 osd.2 514 *** Got signal Terminated *** 2024-03-24T11:43:39.210 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:38 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2[53906]: 2024-03-24T11:43:38.923+0000 7fd5f7d43700 -1 osd.2 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:43:44.799 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:44 smithi179 podman[209486]: 2024-03-24 11:43:44.523606794 +0000 UTC m=+5.718823636 container died c3cd9273cb6cfba0d35a01520a8b6fd6efd76199c227700839cd9a27607efea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:43:45.158 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:44 smithi179 podman[209486]: 2024-03-24 11:43:44.948208117 +0000 UTC m=+6.143424967 container cleanup c3cd9273cb6cfba0d35a01520a8b6fd6efd76199c227700839cd9a27607efea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T11:43:45.158 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:44 smithi179 bash[209486]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:43:45.459 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:45 smithi179 podman[209504]: 2024-03-24 11:43:45.158356842 +0000 UTC m=+0.629152326 container remove c3cd9273cb6cfba0d35a01520a8b6fd6efd76199c227700839cd9a27607efea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-24T11:43:45.829 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:45 smithi179 podman[209596]: 2024-03-24 11:43:45.475604134 +0000 UTC m=+0.019038093 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:43:45.829 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:45 smithi179 podman[209596]: 2024-03-24 11:43:45.592798011 +0000 UTC m=+0.136231966 container create c96be70a7ffa752c19652700170326f3f0c3f9c0c8c909a1972ce2290f6e7d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T11:43:45.829 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:45 smithi179 podman[209596]: 2024-03-24 11:43:45.82889436 +0000 UTC m=+0.372328319 container init c96be70a7ffa752c19652700170326f3f0c3f9c0c8c909a1972ce2290f6e7d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-03-24T11:43:46.129 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:45 smithi179 podman[209596]: 2024-03-24 11:43:45.832381317 +0000 UTC m=+0.375815285 container start c96be70a7ffa752c19652700170326f3f0c3f9c0c8c909a1972ce2290f6e7d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:43:46.129 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:45 smithi179 podman[209596]: 2024-03-24 11:43:45.920678087 +0000 UTC m=+0.464112046 container attach c96be70a7ffa752c19652700170326f3f0c3f9c0c8c909a1972ce2290f6e7d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:43:46.407 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 24 11:43:46 smithi179 podman[209596]: 2024-03-24 11:43:46.128959896 +0000 UTC m=+0.672393856 container died c96be70a7ffa752c19652700170326f3f0c3f9c0c8c909a1972ce2290f6e7d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T11:43:46.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T11:43:46.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:46.864 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T11:43:46.864 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T11:43:46.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.3 2024-03-24T11:43:47.203 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:46 smithi179 systemd[1]: Stopping Ceph osd.3 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:47.402 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:43:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:43:46.985Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.179:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.179:8765: connect: connection refused" 2024-03-24T11:43:47.402 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:43:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:43:46.985Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.179:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.179:8765: connect: connection refused" 2024-03-24T11:43:47.402 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:43:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:43:46.987Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.179:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.179:8765: connect: connection refused" 2024-03-24T11:43:47.402 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:43:46 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:43:46.989Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.179:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.179:8765: connect: connection refused" 2024-03-24T11:43:47.459 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3[59886]: 2024-03-24T11:43:47.203+0000 7fdb60184700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:47.459 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3[59886]: 2024-03-24T11:43:47.203+0000 7fdb60184700 -1 osd.3 514 *** Got signal Terminated *** 2024-03-24T11:43:47.459 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:47 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3[59886]: 2024-03-24T11:43:47.203+0000 7fdb60184700 -1 osd.3 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:43:48.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:48 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:48.707+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:50.152 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:49 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:49.665+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:50.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:50 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:50.620+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:51.274 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:51 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:51.011+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:51.613 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:51 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:51.273+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:51.709 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:51 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3[59886]: 2024-03-24T11:43:51.358+0000 7fdb5c778700 -1 osd.3 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.314986+0000 front 2024-03-24T11:43:26.314926+0000 (oldest deadline 2024-03-24T11:43:51.015109+0000) 2024-03-24T11:43:51.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:51 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:51.611+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:51.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:51 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:51.733+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:52.306 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:52 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:52.000+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:52.640 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:52 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:52.305+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:52.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:52 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:52.727+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:52.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:52 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:52.639+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:52.960 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:52 smithi179 podman[209711]: 2024-03-24 11:43:52.66798862 +0000 UTC m=+5.577209241 container died 23f836912d228a869577dbf19b8567cbbf17233bc332522d2d0204d963df40a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T11:43:53.329 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:52 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:52.957+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:53.405 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:53 smithi179 podman[209711]: 2024-03-24 11:43:53.162240319 +0000 UTC m=+6.071460995 container cleanup 23f836912d228a869577dbf19b8567cbbf17233bc332522d2d0204d963df40a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T11:43:53.405 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:53 smithi179 bash[209711]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3 2024-03-24T11:43:53.651 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:53.329+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:53.709 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:53 smithi179 podman[209730]: 2024-03-24 11:43:53.405367481 +0000 UTC m=+0.729115026 container remove 23f836912d228a869577dbf19b8567cbbf17233bc332522d2d0204d963df40a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:43:53.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:53.699+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:53.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:53.650+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:54.059 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:53 smithi179 podman[209821]: 2024-03-24 11:43:53.639301769 +0000 UTC m=+0.017642536 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:43:54.059 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:53 smithi179 podman[209821]: 2024-03-24 11:43:53.739943537 +0000 UTC m=+0.118284304 container create 6f260cf28c54e0b06664d301b92a2b511bba67862e8fa61b7a2595ca74058138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T11:43:54.059 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:53 smithi179 podman[209821]: 2024-03-24 11:43:53.969191102 +0000 UTC m=+0.347531869 container init 6f260cf28c54e0b06664d301b92a2b511bba67862e8fa61b7a2595ca74058138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T11:43:54.059 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:53 smithi179 podman[209821]: 2024-03-24 11:43:53.972470845 +0000 UTC m=+0.350811613 container start 6f260cf28c54e0b06664d301b92a2b511bba67862e8fa61b7a2595ca74058138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T11:43:54.059 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:54 smithi179 podman[209821]: 2024-03-24 11:43:54.059204276 +0000 UTC m=+0.437545055 container attach 6f260cf28c54e0b06664d301b92a2b511bba67862e8fa61b7a2595ca74058138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T11:43:54.286 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:53 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:53.932+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:54.459 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:54 smithi179 podman[209821]: 2024-03-24 11:43:54.264747443 +0000 UTC m=+0.643088211 container died 6f260cf28c54e0b06664d301b92a2b511bba67862e8fa61b7a2595ca74058138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T11:43:54.615 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:54.285+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:54.812 INFO:journalctl@ceph.osd.3.smithi179.stdout:Mar 24 11:43:54 smithi179 podman[209847]: 2024-03-24 11:43:54.79632153 +0000 UTC m=+0.526761263 container remove 6f260cf28c54e0b06664d301b92a2b511bba67862e8fa61b7a2595ca74058138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T11:43:54.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T11:43:54.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:43:54.863 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T11:43:54.864 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T11:43:54.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.4 2024-03-24T11:43:54.895 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:54.615+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:54.896 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:54.678+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:55.152 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:54 smithi181 systemd[1]: Stopping Ceph osd.4 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:43:55.152 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:54 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:54.936+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:55.647 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:55 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:55.243+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:55.647 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:55 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:55.171+0000 7fabb6508700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:43:55.648 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:55 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:55.171+0000 7fabb6508700 -1 osd.4 514 *** Got signal Terminated *** 2024-03-24T11:43:55.648 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:55 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:55.171+0000 7fabb6508700 -1 osd.4 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:43:55.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:55 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:55.646+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:55.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:55 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:55.659+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:56.230 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:55 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:55.947+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:56.616 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:56 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:56.229+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:56.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:56 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:56.616+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:56.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:56 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:56.676+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:57.243 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:56 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:56.973+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:57.593 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:57 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:57.242+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:57.593 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:57 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:57.242+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:43:57.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:57 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:57.592+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:57.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:57 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:57.651+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:58.225 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:57 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:57.972+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:58.225 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:57 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:57.972+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:34.779193+0000 front 2024-03-24T11:43:34.779031+0000 (oldest deadline 2024-03-24T11:43:57.078899+0000) 2024-03-24T11:43:58.603 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:58 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:58.224+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:58.603 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:58 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:58.224+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:43:58.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:58 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:58.602+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:58.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:58 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:58.662+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:58.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:58 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:58.662+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:43:59.224 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:58 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:58.933+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:43:59.224 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:58 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:58.933+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:34.779193+0000 front 2024-03-24T11:43:34.779031+0000 (oldest deadline 2024-03-24T11:43:57.078899+0000) 2024-03-24T11:43:59.613 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:59.223+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:43:59.613 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:43:59.223+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:43:59.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:59.629+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:43:59.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:43:59.629+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:43:59.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:59.612+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:43:59.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:43:59.612+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:00.236 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:59.973+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:25.378125+0000 front 2024-03-24T11:43:25.378257+0000 (oldest deadline 2024-03-24T11:43:50.678092+0000) 2024-03-24T11:44:00.236 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:43:59 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4[38881]: 2024-03-24T11:43:59.973+0000 7fabaeaf4700 -1 osd.4 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:34.779193+0000 front 2024-03-24T11:43:34.779031+0000 (oldest deadline 2024-03-24T11:43:57.078899+0000) 2024-03-24T11:44:00.633 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:00 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:00.235+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:00.634 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:00 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:00.235+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:00.902 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:00 smithi181 podman[155015]: 2024-03-24 11:44:00.7306721 +0000 UTC m=+5.670390935 container died 0f6127e2f1a16fa4a5651cdc4ca493a26241943a7cafff0eaa2ba32e0c995116 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T11:44:00.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:00 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:00.632+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:00.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:00 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:00.632+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:00.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:00 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:00.671+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:00.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:00 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:00.671+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:01.381 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:01 smithi181 podman[155015]: 2024-03-24 11:44:01.088333693 +0000 UTC m=+6.028052534 container cleanup 0f6127e2f1a16fa4a5651cdc4ca493a26241943a7cafff0eaa2ba32e0c995116 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T11:44:01.382 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:01 smithi181 bash[155015]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4 2024-03-24T11:44:01.382 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:01 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:01.235+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:01.382 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:01 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:01.235+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:01.652 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:01 smithi181 podman[155032]: 2024-03-24 11:44:01.381477942 +0000 UTC m=+0.642281251 container remove 0f6127e2f1a16fa4a5651cdc4ca493a26241943a7cafff0eaa2ba32e0c995116 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:44:01.652 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:01 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:01.617+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:01.652 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:01 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:01.617+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:01.652 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:01 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:01.625+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:01.653 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:01 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:01.625+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:02.002 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:01 smithi181 podman[155125]: 2024-03-24 11:44:01.610692561 +0000 UTC m=+0.018015269 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:44:02.002 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:01 smithi181 podman[155125]: 2024-03-24 11:44:01.749310714 +0000 UTC m=+0.156633421 container create ee975182d27a1410b25e189bccf88cb5eb11dea503026c84da5383db3dba3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:44:02.002 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:02 smithi181 podman[155125]: 2024-03-24 11:44:02.002210818 +0000 UTC m=+0.409533535 container init ee975182d27a1410b25e189bccf88cb5eb11dea503026c84da5383db3dba3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T11:44:02.267 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:02 smithi181 podman[155125]: 2024-03-24 11:44:02.005913009 +0000 UTC m=+0.413235718 container start ee975182d27a1410b25e189bccf88cb5eb11dea503026c84da5383db3dba3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T11:44:02.267 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:02 smithi181 podman[155125]: 2024-03-24 11:44:02.085362367 +0000 UTC m=+0.492685075 container attach ee975182d27a1410b25e189bccf88cb5eb11dea503026c84da5383db3dba3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T11:44:02.647 INFO:journalctl@ceph.osd.4.smithi181.stdout:Mar 24 11:44:02 smithi181 podman[155125]: 2024-03-24 11:44:02.360654051 +0000 UTC m=+0.767976771 container died ee975182d27a1410b25e189bccf88cb5eb11dea503026c84da5383db3dba3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T11:44:02.649 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:02 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:02.273+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:02.649 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:02 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:02.273+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:02.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:02 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:02.660+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:02.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:02 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:02.661+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:02.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:02 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:02.657+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:02.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:02 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:02.657+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:02.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T11:44:03.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:44:03.039 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T11:44:03.039 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T11:44:03.040 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.5 2024-03-24T11:44:03.234 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:03 smithi181 systemd[1]: Stopping Ceph osd.5 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:03.645 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:03.239+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:03.645 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:03.239+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:03.645 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:03.391+0000 7fb2d657c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:44:03.645 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:03.391+0000 7fb2d657c700 -1 osd.5 514 *** Got signal Terminated *** 2024-03-24T11:44:03.645 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:03.391+0000 7fb2d657c700 -1 osd.5 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:44:03.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:03.656+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:03.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:03.656+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:03.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:03.644+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:03.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:03 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:03.644+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:04.597 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:04.275+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:04.598 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:04.275+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:04.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:04.640+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:04.903 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:04.640+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:04.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:04.596+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:04.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:04 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:04.596+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:05.602 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:05.298+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:05.603 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:05.298+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:05.603 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:05.298+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:05.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:05.654+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:05.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:05.654+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:05.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:05.601+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:05.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:05 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:05.601+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:06.560 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:06 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:06.259+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:06.560 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:06 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:06.259+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:06.560 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:06 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:06.259+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:06.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:06 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:06.559+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:06.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:06 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:06.559+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:06.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:06 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:06.631+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:06.903 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:06 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:06.631+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:07.583 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:07.212+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:07.583 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:07.212+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:07.583 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:07.212+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:07.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:07.582+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:07.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:07.582+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:07.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:07.582+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:07.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:07.607+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:22.770672+0000 front 2024-03-24T11:43:22.770538+0000 (oldest deadline 2024-03-24T11:43:48.070568+0000) 2024-03-24T11:44:07.903 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:07.607+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.672312+0000 front 2024-03-24T11:43:35.672223+0000 (oldest deadline 2024-03-24T11:43:59.172184+0000) 2024-03-24T11:44:07.903 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:07 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5[44834]: 2024-03-24T11:44:07.607+0000 7fb2ceb68700 -1 osd.5 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.573608+0000 front 2024-03-24T11:43:42.573630+0000 (oldest deadline 2024-03-24T11:44:06.673474+0000) 2024-03-24T11:44:08.566 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:08 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:08.235+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:08.567 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:08 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:08.235+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:08.567 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:08 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:08.235+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:08.827 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:08 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:08.566+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:08.827 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:08 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:08.566+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:08.827 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:08 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:08.566+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:09.135 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:08 smithi181 podman[155241]: 2024-03-24 11:44:08.827463768 +0000 UTC m=+5.556288361 container died 3e539c97c27ce344ee2ab9b37f53ad91f2deaaa211bb2ccb78b336bbd851bff0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T11:44:09.402 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:09 smithi181 podman[155241]: 2024-03-24 11:44:09.233898282 +0000 UTC m=+5.962722875 container cleanup 3e539c97c27ce344ee2ab9b37f53ad91f2deaaa211bb2ccb78b336bbd851bff0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:09.402 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:09 smithi181 bash[155241]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5 2024-03-24T11:44:09.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:09 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:09.225+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:09.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:09 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:09.225+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:09.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:09 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:09.225+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:09.803 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:09 smithi181 podman[155258]: 2024-03-24 11:44:09.510512944 +0000 UTC m=+0.677479410 container remove 3e539c97c27ce344ee2ab9b37f53ad91f2deaaa211bb2ccb78b336bbd851bff0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T11:44:09.803 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:09 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:09.558+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:09.803 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:09 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:09.558+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:09.803 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:09 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:09.558+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:10.114 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:09 smithi181 podman[155349]: 2024-03-24 11:44:09.803033633 +0000 UTC m=+0.112080318 container create 1c1d6380bb8336107caab2cc9b9a13a1bd3eaa22190712dc3f285ac53ba64a8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:10.114 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:09 smithi181 podman[155349]: 2024-03-24 11:44:09.710594161 +0000 UTC m=+0.019640860 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:44:10.114 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:10 smithi181 podman[155349]: 2024-03-24 11:44:10.03046509 +0000 UTC m=+0.339511783 container init 1c1d6380bb8336107caab2cc9b9a13a1bd3eaa22190712dc3f285ac53ba64a8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:10.114 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:10 smithi181 podman[155349]: 2024-03-24 11:44:10.033548442 +0000 UTC m=+0.342595119 container start 1c1d6380bb8336107caab2cc9b9a13a1bd3eaa22190712dc3f285ac53ba64a8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:44:10.114 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:10 smithi181 podman[155349]: 2024-03-24 11:44:10.113910755 +0000 UTC m=+0.422957444 container attach 1c1d6380bb8336107caab2cc9b9a13a1bd3eaa22190712dc3f285ac53ba64a8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:10.402 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 24 11:44:10 smithi181 podman[155349]: 2024-03-24 11:44:10.327389229 +0000 UTC m=+0.636435916 container died 1c1d6380bb8336107caab2cc9b9a13a1bd3eaa22190712dc3f285ac53ba64a8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T11:44:10.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:10 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:10.199+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:10.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:10 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:10.199+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:10.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:10 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:10.199+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:10.876 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:10 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:10.560+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:10.876 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:10 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:10.560+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:10.876 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:10 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:10.560+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:10.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T11:44:10.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:44:10.956 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T11:44:10.956 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T11:44:10.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.6 2024-03-24T11:44:11.152 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:11 smithi181 systemd[1]: Stopping Ceph osd.6 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:11.556 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:11.203+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:11.556 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:11.204+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:11.556 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:11.204+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:11.556 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:11.295+0000 7f0a891d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:44:11.556 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:11.295+0000 7f0a891d6700 -1 osd.6 514 *** Got signal Terminated *** 2024-03-24T11:44:11.556 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:11.295+0000 7f0a891d6700 -1 osd.6 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:44:11.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:11.555+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:11.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:11.555+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:11.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:11 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:11.555+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:12.598 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:12 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:12.197+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:12.598 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:12 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:12.197+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:12.598 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:12 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:12.197+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:12.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:12 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:12.597+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:12.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:12 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:12.597+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:12.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:12 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:12.597+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:13.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:13.150+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:13.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:13.150+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:13.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:13.150+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:13.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:13.553+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:13.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:13.553+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:13.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:13.553+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:13.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:13 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:13.553+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:14.590 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:14.193+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:14.590 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:14.193+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:14.591 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:14.193+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:14.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:14.590+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:14.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:14.590+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:14.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:14.590+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:14.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:14 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:14.590+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:15.605 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:15.170+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:15.605 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:15.170+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:15.605 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:15.170+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:15.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:15.604+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:15.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:15.604+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:15.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:15.604+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:15.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:15 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:15.604+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:16.569 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:16.151+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.878906+0000 front 2024-03-24T11:43:26.879058+0000 (oldest deadline 2024-03-24T11:43:50.978903+0000) 2024-03-24T11:44:16.569 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:16.151+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:30.979403+0000 front 2024-03-24T11:43:30.979499+0000 (oldest deadline 2024-03-24T11:43:56.879361+0000) 2024-03-24T11:44:16.569 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:16.151+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:42.680680+0000 front 2024-03-24T11:43:42.680742+0000 (oldest deadline 2024-03-24T11:44:04.980630+0000) 2024-03-24T11:44:16.569 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6[50528]: 2024-03-24T11:44:16.151+0000 7f0a817c2700 -1 osd.6 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:49.681738+0000 front 2024-03-24T11:43:49.681668+0000 (oldest deadline 2024-03-24T11:44:16.081887+0000) 2024-03-24T11:44:16.902 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:16 smithi181 podman[155456]: 2024-03-24 11:44:16.74590275 +0000 UTC m=+5.566610255 container died 8a2a45bc7a25c97886abbb2b0651404a1833345213eaa0bea0092bbabfb4b7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:44:16.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:16.568+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:16.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:16.568+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:16.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:16.568+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:16.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:16 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:16.568+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:17.379 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 podman[155456]: 2024-03-24 11:44:17.095295176 +0000 UTC m=+5.916002681 container cleanup 8a2a45bc7a25c97886abbb2b0651404a1833345213eaa0bea0092bbabfb4b7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307) 2024-03-24T11:44:17.380 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 bash[155456]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6 2024-03-24T11:44:17.652 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 podman[155475]: 2024-03-24 11:44:17.3798778 +0000 UTC m=+0.634935026 container remove 8a2a45bc7a25c97886abbb2b0651404a1833345213eaa0bea0092bbabfb4b7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T11:44:17.652 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:17 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:17.609+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:17.652 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:17 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:17.610+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:17.652 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:17 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:17.610+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:17.652 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:17 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:17.610+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:17.992 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 podman[155568]: 2024-03-24 11:44:17.606222149 +0000 UTC m=+0.017892945 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:44:17.992 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 podman[155568]: 2024-03-24 11:44:17.722600666 +0000 UTC m=+0.134271449 container create 65e4c9ff30f072082395ab6772cc42789643faa69cbaa35a1e633dbe04caa063 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T11:44:17.992 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 podman[155568]: 2024-03-24 11:44:17.916961764 +0000 UTC m=+0.328632557 container init 65e4c9ff30f072082395ab6772cc42789643faa69cbaa35a1e633dbe04caa063 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:44:17.992 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 podman[155568]: 2024-03-24 11:44:17.920101766 +0000 UTC m=+0.331772557 container start 65e4c9ff30f072082395ab6772cc42789643faa69cbaa35a1e633dbe04caa063 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T11:44:17.992 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:17 smithi181 podman[155568]: 2024-03-24 11:44:17.991854215 +0000 UTC m=+0.403525012 container attach 65e4c9ff30f072082395ab6772cc42789643faa69cbaa35a1e633dbe04caa063 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T11:44:18.402 INFO:journalctl@ceph.osd.6.smithi181.stdout:Mar 24 11:44:18 smithi181 podman[155568]: 2024-03-24 11:44:18.205635837 +0000 UTC m=+0.617306619 container died 65e4c9ff30f072082395ab6772cc42789643faa69cbaa35a1e633dbe04caa063 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD) 2024-03-24T11:44:18.753 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:18 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:18.599+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:18.754 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:18 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:18.599+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:18.754 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:18 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:18.599+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:18.754 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:18 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:18.599+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:18.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T11:44:18.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:44:18.821 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T11:44:18.821 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T11:44:18.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.7 2024-03-24T11:44:19.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:18 smithi181 systemd[1]: Stopping Ceph osd.7 for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:19.444 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:19.156+0000 7f14ca407700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:44:19.444 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:19.156+0000 7f14ca407700 -1 osd.7 514 *** Got signal Terminated *** 2024-03-24T11:44:19.444 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:19.156+0000 7f14ca407700 -1 osd.7 514 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:44:19.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:19.649+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:19.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:19.649+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:19.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:19.649+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:19.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:19 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:19.649+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:20.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:20.632+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:20.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:20.632+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:20.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:20.632+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:20.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:20 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:20.632+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:21.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:21.630+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:21.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:21.630+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:21.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:21.630+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:21.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:21.630+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:21.903 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:21 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:21.630+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.181:6804 osd.4 since back 2024-03-24T11:43:58.676164+0000 front 2024-03-24T11:43:58.676262+0000 (oldest deadline 2024-03-24T11:44:21.576198+0000) 2024-03-24T11:44:23.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:22 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:22.668+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:23.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:22 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:22.668+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:23.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:22 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:22.668+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:23.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:22 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:22.668+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:23.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:22 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:22.668+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.181:6804 osd.4 since back 2024-03-24T11:43:58.676164+0000 front 2024-03-24T11:43:58.676262+0000 (oldest deadline 2024-03-24T11:44:21.576198+0000) 2024-03-24T11:44:24.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:23.659+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6806 osd.0 since back 2024-03-24T11:43:26.370134+0000 front 2024-03-24T11:43:26.370164+0000 (oldest deadline 2024-03-24T11:43:51.069965+0000) 2024-03-24T11:44:24.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:23.659+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6814 osd.1 since back 2024-03-24T11:43:35.071476+0000 front 2024-03-24T11:43:35.071678+0000 (oldest deadline 2024-03-24T11:43:58.471831+0000) 2024-03-24T11:44:24.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:23.659+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6822 osd.2 since back 2024-03-24T11:43:41.174140+0000 front 2024-03-24T11:43:41.174088+0000 (oldest deadline 2024-03-24T11:44:07.073998+0000) 2024-03-24T11:44:24.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:23.659+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.179:6830 osd.3 since back 2024-03-24T11:43:48.174896+0000 front 2024-03-24T11:43:48.174839+0000 (oldest deadline 2024-03-24T11:44:12.775314+0000) 2024-03-24T11:44:24.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:23 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7[56266]: 2024-03-24T11:44:23.659+0000 7f14c29f3700 -1 osd.7 514 heartbeat_check: no reply from 172.21.15.181:6804 osd.4 since back 2024-03-24T11:43:58.676164+0000 front 2024-03-24T11:43:58.676262+0000 (oldest deadline 2024-03-24T11:44:21.576198+0000) 2024-03-24T11:44:25.014 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:24 smithi181 podman[155685]: 2024-03-24 11:44:24.735409296 +0000 UTC m=+5.681819561 container died 5f5bb49094048f7fcd364cdb38b02bfd18775e54abffdb9a955b60b64e52d135 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-24T11:44:25.015 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 podman[155685]: 2024-03-24 11:44:25.014593011 +0000 UTC m=+5.961003278 container cleanup 5f5bb49094048f7fcd364cdb38b02bfd18775e54abffdb9a955b60b64e52d135 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:44:25.265 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 bash[155685]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7 2024-03-24T11:44:25.628 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 podman[155705]: 2024-03-24 11:44:25.265629523 +0000 UTC m=+0.525542151 container remove 5f5bb49094048f7fcd364cdb38b02bfd18775e54abffdb9a955b60b64e52d135 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-24T11:44:25.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 podman[155800]: 2024-03-24 11:44:25.531272572 +0000 UTC m=+0.017912725 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:44:25.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 podman[155800]: 2024-03-24 11:44:25.633138194 +0000 UTC m=+0.119778328 container create 52077bcf7fd64a1a5727cecc094d37524c0a659d5d8b4ff11b7613760fda9e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T11:44:25.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 podman[155800]: 2024-03-24 11:44:25.852346968 +0000 UTC m=+0.338987114 container init 52077bcf7fd64a1a5727cecc094d37524c0a659d5d8b4ff11b7613760fda9e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T11:44:25.902 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 podman[155800]: 2024-03-24 11:44:25.855423284 +0000 UTC m=+0.342063421 container start 52077bcf7fd64a1a5727cecc094d37524c0a659d5d8b4ff11b7613760fda9e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:26.397 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:25 smithi181 podman[155800]: 2024-03-24 11:44:25.934574632 +0000 UTC m=+0.421214780 container attach 52077bcf7fd64a1a5727cecc094d37524c0a659d5d8b4ff11b7613760fda9e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T11:44:26.397 INFO:journalctl@ceph.osd.7.smithi181.stdout:Mar 24 11:44:26 smithi181 podman[155800]: 2024-03-24 11:44:26.148053081 +0000 UTC m=+0.634693220 container died 52077bcf7fd64a1a5727cecc094d37524c0a659d5d8b4ff11b7613760fda9e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:44:26.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T11:44:26.714 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:44:26.714 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T11:44:26.714 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-24T11:44:26.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-24T11:44:27.005 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:26 smithi179 systemd[1]: Stopping Ceph rgw.foo.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:27.260 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:27 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a[64575]: 2024-03-24T11:44:27.004+0000 7fc12f595700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:44:27.261 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:27 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a[64575]: 2024-03-24T11:44:27.004+0000 7fc13de05ac0 -1 shutting down 2024-03-24T11:44:37.379 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:37 smithi179 bash[209926]: time="2024-03-24T11:44:37Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-24T11:44:37.380 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:37 smithi179 podman[209926]: 2024-03-24 11:44:37.034041659 +0000 UTC m=+10.124782586 container died bec3ddf95fe80ca71f44af4efbaa227d4d746b08116e31afa4c91e121bb07c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-24T11:44:37.633 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:37 smithi179 podman[209926]: 2024-03-24 11:44:37.612457729 +0000 UTC m=+10.703198657 container cleanup bec3ddf95fe80ca71f44af4efbaa227d4d746b08116e31afa4c91e121bb07c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T11:44:37.633 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:37 smithi179 bash[209926]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-24T11:44:37.959 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:37 smithi179 podman[209953]: 2024-03-24 11:44:37.897180833 +0000 UTC m=+0.863105772 container remove bec3ddf95fe80ca71f44af4efbaa227d4d746b08116e31afa4c91e121bb07c0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T11:44:37.959 INFO:journalctl@ceph.rgw.foo.a.smithi179.stdout:Mar 24 11:44:37 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T11:44:38.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-24T11:44:38.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:44:38.085 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-24T11:44:38.086 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T11:44:38.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-6430538c-e9cf-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T11:44:38.402 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 systemd[1]: Stopping Ceph prometheus.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:38.686 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.415Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:44:38.686 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.416Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.416Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.416Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.416Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.416Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.416Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.416Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.424Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.425Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.425Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a[70319]: ts=2024-03-24T11:44:38.425Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T11:44:38.687 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 podman[155905]: 2024-03-24 11:44:38.436600104 +0000 UTC m=+0.121277981 container died e7cb44f2b5d6e78c0b9f3fb922d1f0ccdcc6e3302b34970b37a0e3c128308253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:39.083 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 podman[155905]: 2024-03-24 11:44:38.798885384 +0000 UTC m=+0.483563261 container cleanup e7cb44f2b5d6e78c0b9f3fb922d1f0ccdcc6e3302b34970b37a0e3c128308253 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:39.083 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Mar 24 11:44:38 smithi181 bash[155905]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:44:39.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6430538c-e9cf-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T11:44:39.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:44:39.223 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T11:44:39.223 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:44:39.415 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:44:41.959 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:41 smithi179 systemd[1]: Stopping Ceph alertmanager.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:42.249 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:41 smithi179 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a[81675]: ts=2024-03-24T11:44:41.968Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:44:42.249 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:42 smithi179 podman[210431]: 2024-03-24 11:44:42.00924106 +0000 UTC m=+0.134198257 container died 12bd225c6028c54e350563a65647455f2c371ae8a0c5fec3f14eb54f12daf054 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:42.853 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:42 smithi179 podman[210431]: 2024-03-24 11:44:42.585881206 +0000 UTC m=+0.710838447 container cleanup 12bd225c6028c54e350563a65647455f2c371ae8a0c5fec3f14eb54f12daf054 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:42.854 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:42 smithi179 bash[210431]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:44:43.135 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:42 smithi179 podman[210445]: 2024-03-24 11:44:42.853975214 +0000 UTC m=+0.845060863 container remove 12bd225c6028c54e350563a65647455f2c371ae8a0c5fec3f14eb54f12daf054 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:43.135 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:42 smithi179 podman[210445]: 2024-03-24 11:44:42.938332657 +0000 UTC m=+0.929418303 volume remove 1c5bb53b4443c6be6d0b3aed2597725e1df146bc61b12ddef09ec941f6673f78 2024-03-24T11:44:43.460 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:43 smithi179 systemd[1]: Stopping Ceph node-exporter.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:43.460 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:43 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:44:43.460 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:43 smithi179 systemd[1]: Stopped Ceph alertmanager.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:44:43.460 INFO:journalctl@ceph.alertmanager.a.smithi179.stdout:Mar 24 11:44:43 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.720s CPU time. 2024-03-24T11:44:43.847 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:43 smithi179 podman[210555]: 2024-03-24 11:44:43.592723071 +0000 UTC m=+0.099987935 container died e4eaeeb17d51cff887b431ded96f15859e9081c9822e812e397380935e85f2ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:44.209 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:44 smithi179 podman[210555]: 2024-03-24 11:44:44.024301576 +0000 UTC m=+0.531566434 container cleanup e4eaeeb17d51cff887b431ded96f15859e9081c9822e812e397380935e85f2ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:44.209 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:44 smithi179 bash[210555]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T11:44:44.579 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:44 smithi179 podman[210569]: 2024-03-24 11:44:44.226053383 +0000 UTC m=+0.637550781 container remove e4eaeeb17d51cff887b431ded96f15859e9081c9822e812e397380935e85f2ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:44:44.579 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:44 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:44:44.579 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:44 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T11:44:44.579 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:44 smithi179 systemd[1]: Stopped Ceph node-exporter.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:44:44.579 INFO:journalctl@ceph.node-exporter.a.smithi179.stdout:Mar 24 11:44:44 smithi179 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 4.821s CPU time. 2024-03-24T11:44:45.123 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:44:45.311 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:44:47.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:47 smithi181 systemd[1]: Stopping Ceph iscsi.iscsi.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:44:47.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:47 smithi181 ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a[61417]: debug Shutdown received 2024-03-24T11:44:57.846 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:57 smithi181 bash[156313]: time="2024-03-24T11:44:57Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-24T11:44:57.846 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:57 smithi181 podman[156313]: 2024-03-24 11:44:57.564378621 +0000 UTC m=+10.114664915 container died e97233b20e3465a70a732b2c00c6f3d694643c8e37a343771dfcf79aa77fcf78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-24T11:44:58.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:57 smithi181 podman[156313]: 2024-03-24 11:44:57.935419513 +0000 UTC m=+10.485705808 container cleanup e97233b20e3465a70a732b2c00c6f3d694643c8e37a343771dfcf79aa77fcf78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:44:58.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:57 smithi181 bash[156313]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a 2024-03-24T11:44:58.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:58 smithi181 podman[156340]: 2024-03-24 11:44:58.245156424 +0000 UTC m=+0.668170855 container remove e97233b20e3465a70a732b2c00c6f3d694643c8e37a343771dfcf79aa77fcf78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:44:58.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:58 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T11:44:58.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:58 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-24T11:44:58.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:58 smithi181 systemd[1]: Stopped Ceph iscsi.iscsi.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:44:58.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Mar 24 11:44:58 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Consumed 3.437s CPU time. 2024-03-24T11:45:00.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:44:59 smithi181 systemd[1]: Stopping Ceph grafana.a for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:45:00.460 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 podman[156599]: 2024-03-24 11:45:00.154171784 +0000 UTC m=+0.089729740 container died 348827ab325965fa7dcd6bd382b60c0b34934551f0a9843d857d9a27989af10f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.openshift.expose-services=, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , io.buildah.version=1.32.0, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., version=8.5) 2024-03-24T11:45:00.460 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 podman[156599]: 2024-03-24 11:45:00.460237995 +0000 UTC m=+0.395795950 container cleanup 348827ab325965fa7dcd6bd382b60c0b34934551f0a9843d857d9a27989af10f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vendor=Red Hat, Inc., version=8.5, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64) 2024-03-24T11:45:00.853 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 bash[156599]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a 2024-03-24T11:45:00.853 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 podman[156613]: 2024-03-24 11:45:00.669891394 +0000 UTC m=+0.514869377 container remove 348827ab325965fa7dcd6bd382b60c0b34934551f0a9843d857d9a27989af10f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, distribution-scope=public, name=ubi8, vcs-type=git, vendor=Red Hat, Inc., release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-24T11:45:00.853 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:45:01.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:01 smithi181 systemd[1]: Stopping Ceph node-exporter.b for 6430538c-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:45:01.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-24T11:45:01.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 systemd[1]: Stopped Ceph grafana.a for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:45:01.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Mar 24 11:45:00 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@grafana.a.service: Consumed 12.986s CPU time. 2024-03-24T11:45:01.573 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:01 smithi181 podman[156725]: 2024-03-24 11:45:01.300353581 +0000 UTC m=+0.102638416 container died 1f5d71af48a9896e4e9b21c1def1010da9650bbe66c8696a56a8cafb4c54389c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:45:01.899 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:01 smithi181 podman[156725]: 2024-03-24 11:45:01.672864552 +0000 UTC m=+0.475149388 container cleanup 1f5d71af48a9896e4e9b21c1def1010da9650bbe66c8696a56a8cafb4c54389c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:45:01.899 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:01 smithi181 bash[156725]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T11:45:02.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:01 smithi181 podman[156739]: 2024-03-24 11:45:01.899048701 +0000 UTC m=+0.596210992 container remove 1f5d71af48a9896e4e9b21c1def1010da9650bbe66c8696a56a8cafb4c54389c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-6430538c-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:45:02.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:01 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:45:02.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:02 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T11:45:02.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:02 smithi181 systemd[1]: Stopped Ceph node-exporter.b for 6430538c-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:45:02.152 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Mar 24 11:45:02 smithi181 systemd[1]: ceph-6430538c-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 4.445s CPU time. 2024-03-24T11:45:02.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:45:02.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:45:02.862 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T11:45:02.865 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882/remote/smithi179/crash 2024-03-24T11:45:02.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:45:02.927 INFO:teuthology.orchestra.run.smithi179.stderr:tar: /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:45:02.927 INFO:teuthology.orchestra.run.smithi179.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:45:02.929 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882/remote/smithi181/crash 2024-03-24T11:45:02.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:45:02.980 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:45:02.981 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:45:02.982 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T11:45:02.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T11:45:03.070 INFO:teuthology.orchestra.run.smithi179.stdout:2024-03-24T11:30:00.000137+0000 mon.a (mon.0) 1187 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T11:45:03.070 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T11:45:03.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo egrep '\[SEC\]' /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T11:45:03.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo egrep '\[ERR\]' /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T11:45:03.178 DEBUG:teuthology.orchestra.run.smithi179:> sudo egrep '\[WRN\]' /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T11:45:03.246 INFO:teuthology.orchestra.run.smithi179.stdout:2024-03-24T11:30:00.000137+0000 mon.a (mon.0) 1187 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-24T11:45:03.247 INFO:tasks.cephadm:Compressing logs... 2024-03-24T11:45:03.247 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:45:03.289 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:45:03.345 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-24T11:45:03.346 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:45:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T11:45:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:45:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T11:45:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T11:45:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T11:45:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T11:45:03.359 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-24T11:45:03.360 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.5.log: /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:45:03.360 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:45:03.360 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:45:03.360 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T11:45:03.366 INFO:teuthology.orchestra.run.smithi179.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:45:03.366 INFO:teuthology.orchestra.run.smithi179.stderr:: No such file or directory 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr: 57.6% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:45:03.376 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:45:03.377 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T11:45:03.377 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:45:03.377 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T11:45:03.388 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.5% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:45:03.415 INFO:teuthology.orchestra.run.smithi179.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:45:03.472 INFO:teuthology.orchestra.run.smithi179.stderr: 91.2% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:45:03.486 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.b.log: 81.6% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:45:03.504 INFO:teuthology.orchestra.run.smithi181.stderr: 89.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:45:03.587 INFO:teuthology.orchestra.run.smithi181.stderr: 92.4% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T11:45:03.595 INFO:teuthology.orchestra.run.smithi181.stderr: 67.2% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-24T11:45:03.624 INFO:teuthology.orchestra.run.smithi181.stderr: 90.1% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:45:03.625 INFO:teuthology.orchestra.run.smithi181.stderr: 85.0% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:45:03.640 INFO:teuthology.orchestra.run.smithi181.stderr: 91.1% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:45:03.768 INFO:teuthology.orchestra.run.smithi179.stderr: 94.0% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:45:03.769 INFO:teuthology.orchestra.run.smithi179.stderr: 88.9% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:45:04.231 INFO:teuthology.orchestra.run.smithi179.stderr: 89.6% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T11:45:06.360 INFO:teuthology.orchestra.run.smithi181.stderr: 91.2% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T11:45:13.060 INFO:teuthology.orchestra.run.smithi179.stderr: 91.2% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T11:45:22.620 INFO:teuthology.orchestra.run.smithi179.stderr: 90.5% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T11:45:46.554 INFO:teuthology.orchestra.run.smithi181.stderr: 94.3% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T11:47:34.391 INFO:teuthology.orchestra.run.smithi181.stderr: 93.7% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T11:48:23.561 INFO:teuthology.orchestra.run.smithi181.stderr: 93.5% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T11:49:02.017 INFO:teuthology.orchestra.run.smithi181.stderr: 93.6% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T11:49:02.019 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-03-24T11:49:02.019 INFO:teuthology.orchestra.run.smithi181.stderr:real 3m58.716s 2024-03-24T11:49:02.019 INFO:teuthology.orchestra.run.smithi181.stderr:user 4m9.375s 2024-03-24T11:49:02.019 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m10.827s 2024-03-24T11:51:40.482 INFO:teuthology.orchestra.run.smithi179.stderr: 93.5% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T11:56:43.300 INFO:teuthology.orchestra.run.smithi179.stderr: 93.3% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T11:58:04.505 INFO:teuthology.orchestra.run.smithi179.stderr: 92.8% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T11:59:00.479 INFO:teuthology.orchestra.run.smithi179.stderr: 92.8% -- replaced with /var/log/ceph/6430538c-e9cf-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T11:59:00.481 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-24T11:59:00.481 INFO:teuthology.orchestra.run.smithi179.stderr:real 13m57.182s 2024-03-24T11:59:00.481 INFO:teuthology.orchestra.run.smithi179.stderr:user 6m39.704s 2024-03-24T11:59:00.481 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m19.538s 2024-03-24T11:59:00.482 INFO:tasks.cephadm:Archiving logs... 2024-03-24T11:59:00.483 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882/remote/smithi179/log 2024-03-24T11:59:00.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:00:05.796 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882/remote/smithi181/log 2024-03-24T12:00:05.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T12:00:45.107 INFO:tasks.cephadm:Removing cluster... 2024-03-24T12:00:45.107 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 --force 2024-03-24T12:00:45.387 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T12:00:46.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6430538c-e9cf-11ee-95cd-87774f69a715 --force 2024-03-24T12:00:46.352 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: 6430538c-e9cf-11ee-95cd-87774f69a715 2024-03-24T12:00:46.939 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T12:00:46.939 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T12:00:46.962 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T12:00:46.987 INFO:tasks.cephadm:Teardown complete 2024-03-24T12:00:46.988 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T12:00:46.998 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T12:00:46.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T12:00:47.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T12:00:47.123 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T12:00:47.131 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T12:00:47.131 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:00:47.134 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:00:47.145 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:00:47.147 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:00:47.176 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:00:47.176 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-24T12:00:47.176 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +821us[ +819us] +/- 69ms 2024-03-24T12:00:47.176 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 +34us[ +31us] +/- 69ms 2024-03-24T12:00:47.176 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -877us[ -880us] +/- 73ms 2024-03-24T12:00:47.177 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:00:47.189 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:00:47.189 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-03-24T12:00:47.189 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +826us[ +826us] +/- 69ms 2024-03-24T12:00:47.189 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 +51us[ +64us] +/- 69ms 2024-03-24T12:00:47.189 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -880us[ -866us] +/- 73ms 2024-03-24T12:00:47.189 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:00:47.190 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T12:00:47.199 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T12:00:47.199 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T12:00:47.208 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:00:47.318 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T12:00:47.390 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:00:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:00:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:47.448 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T12:00:47.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T12:00:47.466 INFO:teuthology.task.internal:Duration was 3329.652178 seconds 2024-03-24T12:00:47.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T12:00:47.476 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T12:00:47.476 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:00:47.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T12:00:47.543 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:00:47.564 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:00:47.710 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T12:00:47.711 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-03-24T12:00:47.711 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:00:47.731 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-03-24T12:00:47.731 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T12:00:47.769 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T12:00:47.770 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:00:47.773 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T12:00:47.839 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T12:00:47.839 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:00:47.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T12:00:49.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T12:00:49.089 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T12:00:49.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:00:49.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T12:00:49.182 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T12:00:49.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:00:49.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:00:49.229 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-03-24T12:00:49.232 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-03-24T12:00:49.294 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:00:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.349 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T12:00:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:00:49.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T12:00:49.374 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T12:00:49.376 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882/remote/smithi179 2024-03-24T12:00:49.376 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:00:49.455 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882/remote/smithi181 2024-03-24T12:00:49.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:00:49.496 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T12:00:49.497 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:00:49.499 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T12:00:49.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T12:00:49.557 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T12:00:49.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T12:00:49.637 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T12:00:49.638 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:00:49.640 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:00:49.652 INFO:teuthology.orchestra.run.smithi179.stdout: 265421 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:00 /home/ubuntu/cephtest 2024-03-24T12:00:49.654 INFO:teuthology.orchestra.run.smithi181.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 12:00 /home/ubuntu/cephtest 2024-03-24T12:00:49.655 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T12:00:49.667 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T12:00:49.689 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T12:00:49.717 DEBUG:teuthology.nuke:shortname: smithi179 2024-03-24T12:00:49.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:00:49.742 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:56:55.251715', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGjCNhajPt4OOjUC4uIfGqvc5ymEf2cKtxBWraVxdoQw80xQOQ0+Ljj+lis9NYcREX+7b2JsfDZM89W6aIVCts='} 2024-03-24T12:00:49.746 DEBUG:teuthology.nuke:shortname: smithi181 2024-03-24T12:00:49.746 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:00:49.767 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:56:55.250049', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARFtn7MZU/LuZvg4bgTANeO9ccOZvJLbhWtvvvNMk4/e0bWXTXFpmijjuFDaXC7u3kfMsUNxuOcuKZrbFKwFOs='} 2024-03-24T12:00:49.798 INFO:teuthology.orchestra.console.smithi179:Power off 2024-03-24T12:00:49.798 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:00:49.823 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2024-03-24T12:00:49.824 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:00:49.845 INFO:teuthology.orchestra.console.smithi181:Power off 2024-03-24T12:00:49.845 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:00:49.857 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-24T12:00:49.870 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-03-24T12:00:49.870 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:00:49.894 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-03-24T12:00:53.858 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:00:53.895 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:00:53.970 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-24T12:00:54.005 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-03-24T12:00:57.976 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:00:58.006 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:00:58.086 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-24T12:00:58.116 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-03-24T12:00:58.217 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-03-24T12:01:02.090 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:02.200 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2024-03-24T12:01:02.301 INFO:teuthology.orchestra.console.smithi179:Power off completed 2024-03-24T12:01:02.405 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3329.652177810669 failure_reason: '"2024-03-24T11:30:00.000137+0000 mon.a (mon.0) 1187 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-24T12:01:02.405 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T12:01:02.495 INFO:teuthology.run:FAIL